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Abstract 

We give a characterization of a generalized Whittaker model of a de- 
generate principal series representation of GL(n, R) as the kernel of some 
differential operators. By this characterization, we investigate some exam- 
ples on GL(4, R). We obtain the dimensions of the generalized Whittaker 
models and give their basis in terms of hypergeometric functions of one 
and two variables. We show the multiplicity one of the generalized Whit- 
taker models by using the theory of hypergeometric functions. 

1 Introduction 

Our interest in this paper is generalized Whittaker models of degenerate prin- 
cipal representations. There are many studies about them for admissible (non- 
degenerate) characters of unipotent radicals of parabolic subgroups (for example 
[BJ, [TBJ, [25], [3D], [31]). In the case of degenerate principal scries representations, 
Yamashita gives existence theorem and multiplicity formula for wide classes of 
generalized Whittaker models, i.e., generalized Whittaker models for general- 
ized Gelfand-Graev representations in |31| . However, their techniques strongly 
depend on the admissibility of the characters of the unipotent subgroups. On 
the other hand, if we regard the Whittaker models as an analogue of Fourier 
coefficients of an automoprhic form at a cusp, we often meet the necessity to 
consider non-admissible characters. For example, Terras gives an expansion 
of the Epstein zeta function in terms of modified Bessel functions [27]. Non- 
admissible characters play important roles there. The Epstein zeta function 
corresponds to the degenerate principal series representation of GL(n,M) in- 
duced from the character of the maximal parabolic subgroup P\_ n which fixes 
the unit vector e n = (0, . . . , 0, 1) (cf. [TTj). Hence the Fourier coefficients given 
by Terras can be seen as the generalized Whittaker functions for this represen- 
tation. It seems, however, widely open about the problem of the existence and 
the multiplicity formula of the generalized Whittaker models for non-admissible 
characters of unipotent subgroups, though recently the solutions of such prob- 
lems for degenerate characters of maximal unipotent subgroup are obtained by 
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Abe and Oshima independently PQ,[2T] for degenerate principal series repre- 
sentations with generic parameters. In this paper, we will give some examples 
about this problems in the case of GL(4,R). 

The other purpose of this paper is to give an expression of the generalized 
Whittakcr function as a hypergeometric function of several variables. According 
to the recent work of Oshima and Shimeno [53], Whittaker functions can be 
seen as the confluent hypergeometric functions obtained from Heckman-Opdam 
hypergeometric functions. The similarities of Whittaker functions with spherical 
functions were already pointed out by Hashizume in [7]. Also there are various 
explicit pictures of Whittaker functions as hypergeometric functions of several 
variables given by Oda and his collaborators (see [10] for the reference). We 
will show that the generalized Whittaker functions of the degenerate principal 
series representations of GL(4,R) are written by modified Bessel functions and 
Horn's hypergeometric function Hio in this paper. There is a similar work on 
SL(3,R) in Q2|. 

Let us explain the contents of this paper. In Section 2 and Section 3, we will 
give a characterization of a Whittaker model of a degenerate principal series 
representation of GL(n,M.) as the kernel of a family of differential operators. 
More precisely, let G = GL(n,M) and consider an Iwasawa decomposition G = 
KAN where K = 0(n), A is the group of diagonal matrices with positive real 
entries, and N is the group of lower triangular matrices with Is on diagonal 
entries. We take an increasing sequence of positive integers ending at n, i.e., 
= {ni, . . . , rij,} with < n\ < n% < • • • < til = n. Then let Pq be the 
parabolic subgroup corresponding to the sequence 6 and take the Langlands 
decomposition P® = MqAqNq. For a linear mapping A G Hom^Lie^e), C), 
we can consider an induced representation C°° Tndp e (1m s &>e x ® 1n )- We call 
this representation a degenerate principal series representation. The underlying 
representation space of this is 

G°°(G/P0,A) = 

{/ G G°°(G) I f(gp) = (l Me ®e A ® l Ne )(p- l )f(g), g € G,p G P e } 

and the action of G is defined by the left translation. Then we consider an ideal 
of U(g) the universal enveloping algebra of gc = gl(n,C) such that Iq(X) — 
{X G U(q) I R x f = 0, / € G°°(G/Pe,A)}. Here R x is the right derivation 
by X G U(q). We consider A as an element of Homj{(Lie(A), C) and we assume 
it is regular and dominant. Under this assumption, the generators of the ideal 
2e(A) is known by Oshima (cf. Theorem 12. 7[) . Let U be a closed subgroup N 
and (rj, V^) an irreducible unitary representation of U. We consider the space 
C™(U\G) = {/: G - V™ smooth | f{ug) = r]{u)f{g),u G U,g € G} where 
is the space of smooth vectors in V v . Let Ae.A be the Harish-Chandra 
module of G°°(G/ Pq, A) and Xq a its dual Harish-Chandra module, i.e., the 
space of K-fmite vectors of Houic(Xq_\, C). The generalized Whittaker model 
is the image of X& x by the element of Hom gCi ^(Ae,A, Gf°([/\G)). Then we 
can show the following characterization theorem of the generalized Whittaker 
model. 

Theorem 1.1 (see Theorem I3.6j) . Assume that Xq a is irreducible. We take a 
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nonzero K -fixed vector /o in Xq a . Then the following mapping 

$: Hom 0c , K (X* X ,C™(U\G)) ^ C™{U\G/ K,I e {\)) 
W .— > W(/o)(ff) 

is a linear isomorphism. Here 

C™(U\G/K,I e (X)) 

= {f:G^V™ smooth\f(ngk) = n{n)f{g),g£ G,n G U, fc G X 

and i*x/(<?) = 0, XG/ e (A)}. 

This theorem is an analogue of the theorem for the generalized Whittaker 
models of unitary highest weight modules obtained by Yamashita 32], 33J . 

From Section 2J we consider examples on degenerate principal series repre- 
sentations of GL(4, R) induced from characters of maximal parabolic subgroups 
Pi. 4 and P2.4 by using above theorem. We will determine the dimension of 
H.oxn Bc<K (X£ iX ,C™(U\G)) and the basis of C™(U\G/K,I (\)). Let us ex- 
plain more detailed settings. As the space C™(U\G), we consider the space 
defined as follows. 

1. the group U is a closed subgroup of N and 77 is its unitary character, 

2. the unitary induced representation L 2 Tnd^f rj is an irreducible unitary 
representation of N. 

We will classify the G-equivalent classes of these C^(U\G) in Section |4~T1 
(see Proposition 14. 1 0|) . 

There is a linear isomorphism from the space (U\G/K) onto C°°(U\N x 
A) (cf. Lemma f4 . 1 1 [) . In Section l4~2l we will see how the the action of the Lie 
algebra g is written as differential operators on C°°(U\N x A). 

Our main results are in Section 14.31 In this section, we will give the di- 
mensions of C™(U\G/K,Iq(\)) and the basis of them as the functions on 
C°°(J7\A r x A). These basis can be written in terms of modified Bessel func- 
tions and Horn's hypergeometric functions H10 (see Theorem I4.22[ Theorem 
14.251 Theorem f4. 2 71 Theorem f4.28| Theorem f4 29[ Theorem |4.31|) . According 
to these theorems, we can conclude the following. For the degenerate principal 
series representation induced from a character of Pi. 4, the multiplicity one the- 
orem is true for the generalized Whittaker models for characters of the closed 
proper subgroups of N. On the other hand, for the degenerate principal series 
representation induced from a character of P24, the multiplicity one theorem 
is no longer true. This fact seems to correspond to the result of Terras in [2"8] . 
In that paper, she could determine only the nonsingular terms in the Fourier 
expansion of Eisenstein series corresponding to this degenerate principal series 
representation (Theorem 1 in And she could not say anything about 

degenerate terms in Fourier expansion. The multiplicities of the generalized 
Whittaker models corresponding to these Fourier coefficients seems to be one of 
the cause of this phenomenon. 

Finally, we give some facts about Horn's hypergeometric functions in Ap- 
pendix. 
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2 Spherical degenerate principal series represen- 
tations of GL(n, R) 



In this section, we study degenerate principal series representations of GL(n, R) 
and their annihilators in the enveloping algebra U(gl(n,C)). T. Oshima shows 
that the image of a degenerate principal series representation by the Poisson 
transform is characterized by the kernel of the annihilator of the degenerate 
principal series representation [15]. He also give the explicit generators for its 
annihilator [2U] , [52] . We will give a brief review of these results here. 

2.1 Spherical degenerate principal series representations 



Let G = GL(n,M). We denote its Lie algebra by g = g[(n,R). We take the 
Iwasawa decomposition of G as G = KAN, where K = 0(n) and A is the group 
of n x n diagonal matrices with positive real entries and N is the group of lower 
triangular matrices with Is on the diagonal entries. Let be the matrix with 
1 in the (i, j)-entry and elsewhere. We introduce a non-degenerate bilinear 
form on gc = gl(n, C) = M(n, C) by 



By this bilinear form, we identify gc with its dual space g c . The dual basis 
{E*^} of {Eij} is given by E*a — Eji. For simplicity, we write = E*^ 
We consider the Lie algebra 



i=l 

of A. Then the root system of a in g is 

A(fl, a) = {et - e 3 ■ \ 1 < i=£ j < n}. 

We put on — ei+i — e.j for i = 1, . . . , n — 1 and fix a simple system of A(g, a) as 

n(fl, a) = {ai,... ,a„_i}. 

Then the positive system of A(g,o) associated to II(g, a) is A + (g, a) = {e 2 ; — 
Sj | 1 < j < i < n}. Then the Lie algebra n of N is written by 



of GL(n,R). 



(X, Y) = tr(AF) for X, Y G flc . 



n 




n = 



•a 



aeA+(j,a) 



z2 R-Ey 



i>j 
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where $ a = {X E g | &d(H)X = a{H)X for H E a}. On the other hand, let N 
be the group of upper triangular matrices with Is on the diagonal entries. Then 
the Lie algebra ri of N is also written by 

n= ®- a 

aeA+(s,a) 

= ^ ^- E ij ■ 

i<j 

Let O — {ni, . . . , Ul} be a sequence of strictly increasing positive integers ending 
at n, i.e., (0 = n <)n\ < n 2 < ••• < til(= n). For this O, the associated 
standard parabolic subgroup Pq can be defined as follows. Let 



ae = {^2a k E u \ a k ER,k = 1, . . . ,L}. 

fe=l + l 



Let Lq be the centralizer of ae in G, i.e., 



(h 



\ 



\ 



U E GL(rii — 



and le its Lie algebra which is the centralizer of ae in g. We put 



te(*)>teO') 



where 



lq(v) = i if rii-i < v < rii for i = 1, . . . , L. 
The corresponding analytic subgroup of G is N<~, = expne, i.e., 



(2.1) 



N e = 



n = 



(In- 

N 2 1 In> 2 

N31 N 32 I n , 3 

\N L i N L2 N L3 



I Nij E M«,n';I 



> . 



Here I m denotes the identity matrix of size m and M(k, I; K) denotes the 
space of matrices of size k x I with components in M. We also define ne = 

Eie(i)<ie(j) RE V and ^ e = eXpHe aS WelL 

Then we define the parabolic subgroup P® = LqNq, i.e., 



^e = 4 



P = 



* 92 



E GL(n, M) I ft e GL(n 4 - ni_i,l 



> . 
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Its Lie algebra is written as pe = le © ne- 

For (Ai, A 2 , • • • ,Xl) G C l , we define a 1-dimensional representation of P@, 
X: Pe — > C x as follows, 

X(p) = | dct( 3l )| Al | det( 32 )| A2 • • • | det( 5L )| A - , for p G Pe- 

We define a sphcrieal degenerate principal series representation of G, denote by 
= C°°-m.dp q(A). The underlying representation space is 

C°°(G/P e ; A) - {0 G G°°(G) | <j>{gp) = X(p)0(g), geG, P e P@} 

where C°°(G) is the space of C°°-functions on G. The action of G on this 
space is defined by the left translation, 7re \(g)<ft(x) = (j)(g x) for g G G and 
0eG°°(G/Pe;A)'. 

We consider the annihilator of G°°(G/Pe;A) in the universal enveloping 
algebra. Let U(g) be the universal enveloping algebra of Qc- We can see U(g) 
as the ring of left G-invariant differential operators on G°°(G) by the natural 
extent ion of the differentiation of the right translation, 

Rx(f)(g) = j t f(geMtX))\ t =o. 

for X G 0, / G G°°(G). The representation of [/(g) on C°°{G/P @ ] A) is defined 
by the differentiation of ne.\, i.e., for leg, <fi G G°°(G/Pe; A), 7re,A(A")0(ai) = 

^0(exp(-tA>)| t=o . 

Let L s and i? g be the left and right translations by 5 G G respectively i.e., 
£ fl /(s) - fig' 1 *) and P s /(z) - .f(x 5 ) for / G G°°(G). 

Definition 2.1. We define the annihilator of G°°(G/Pe; A) in J7(g) fey 
Ann [/(fl) ( 7 r e ,A) = {Xe t/(fl) ; ttq^X^x) = 0, /or oM 4> G G°°(G/Pe; A)}. 
T/iis is a two-sided ideal ofU(g). 

We consider an antiautomorphism t of U(q) defined by i(XY) = (—Y)(—X) 
for X, Y G 0c- We denote the differentiation of A by o!A: pe - * C Although 
the proposition below is a well-known fact, we give a proof for the completeness 
of the paper. 

Proposition 2.2. The annihilator of ire, \ * s written as follows, 

t(Anii t/ ( fl )(7re,A)) = Pi Ad(,g)J e (rfA). 

gee 

Here 

Je(dX)= ]T U(q)(X - dX(X)) 

Xepe 

is a left ideal ofU(o). 

Proof. For X G pe and / G G°°(G/P e ; A), we have 



^/(.9) = ^/(.g-cxptA)| t=0 
= ^A(exptA)| t=0 /(s). 



(2.2) 
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This implies Rxf = for X E J@(dX). We recall the equation Lxf{g) = 
#Ad( 9 -i) t (x)/(ff), X G U(g). Since X G f| geG Ad(g) J e {dX) implies Ad(g)X G 
Js(dX), we have 

L L (x)f{g) = RAd( g -i)xf(g) = 0, 

for X G PloeG Ad (5) Je(dX). Hence we have the inclusion C\ g eG Ad(g)Je(dX) C 
L(Ann uiB) (Tr 0iX )). 

On the other hand, we take X G Ann[/( fl ) (tto.a), and put X go = Ad(g ( ^ 1 )(,(X) 
for go G G. Then we have 

Rx g J{g) = L gag -i(R Xgo f)(ga) = R Xgo (L gog -i f)(g Q ) ^ 
= L x (Lg Qg -if)(g ) = L x (Tre,x(g Q g^ 1 )f)(go) = 

for / G C°°(G/Pe; A). By the decomposition g = ne CD pe and the Poincare- 
Birkoff-Witt theorem, we have 

f/( ) = U(n @ ) J e (dX) 

where U(ri@) is the universal enveloping algebra of ne <8>m C. Hence there exist 
Y G 17 (ne) and Z G Je(dA) such that X go =Y + Z. By the equation {Zg) , we 
have = for 5 G G and / G G°°(G/P e ; A). Therefore the equation (jOj) 

lead us that = Rx go f(g) — Ryf{g)- We will show V = 0. Then this means 
X go G Je(dX). Therefore we can show the inclusion f) eG Ad(g)J&(dX) C 
t(Ann [/(£ , ) (7r e ,A))- 

We consider the space of compactly supported G°°-functions on TVe, and 
denote it by G£°(7Ve). For g G N e P e , we take n(e?) G 7V e and G P e such 
that 5 = n(g)p(g). Then we have an injection 

G °°(lVe) — G°°(G/P e ;A) _ 

f ^ \Kp(g))f(n(g)) iigeN e P e 
1 otherwise. 

By this injection, we can consider C^°(Nq) C G°° (G/ Pq; A). Therefore if we 
recall that Ryf{g) = for g G G, / G G°°(G/Pe; A), we have 

fly/(n) = for neNeJe G °°(F e ). 

For any V> G C°°(Nq) and n G iVe, there exists / G G£°(7Ve) such that tp = f 
on some neighbourhood of n in N®. Hence this implies 

R Y ip(fi) = for n G iVe, V € G°°(iVe)- 

Therefore Y G f7(tle) must be 0, because of the fact that U(fi&) is identified with 
the ring of all left invariant differential operators in Nq. Hence X go G Je(A) 
for any go G G. This complete the proof. □ 

2.2 The Poisson transform for the degenerate principal 
series representation. 

For simplicity we write Iq(X) — C\ gE Q Ad(g)jQ(dX). Then we will see that this 
ideal 7e (A) characterizes the image of the Poisson transform from the degenerate 
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principal series. To explain this fact, we should extend the representation space 
of the degenerate principal series to the space of hyperfunctions on G. 

The space B{G) of hyperfunctions on G is a left G-module by the left trans- 
lation G x 13(G) 3 (g,f(x)) i— ► f(g~ 1 x). We take a parabolic subgroup Pq of 
G. Also we take a character A: Pq — > C x for (Ai, • • • , Xl) € C L . Then we can 
define a G-submodule 

B(G/P e ; A) = {/ e 6(G) | /(zp) = A(p)/(z) for p G P e }, 

as in Section [5TTJ Let M = {k € K \ kak^ 1 = a, a £ A}, then we can define the 
minimal parabolic subgroup P = P{i,2,---,n} = MAN. We define a character 
of Pa by 

A e : P — > C x 

L "i+i 

man i — ► J^J ]Q a**, 

i— 1 

for to g M, a G A, n € iV. Now we introduce the Poisson transform of 
B(G/P ;Xe). 

Definition 2.3. The Poisson transform is a G-homomorphism 

V x : B(G/P ;X e ) — > B(G/iQ 

/ i — ► F(x) = / f(xk)dk, xeG. 
Jk 

Here dk is the normalized Haar measure on K so that J K dk — 1 . 

We define a character of the center Z(g) of U(g). Let <iAe : Lie(P ) — * C 
be the differentiation of Ae- By the restriction to a C Lie(P Q ), we can regard 
dX(j G a£. Let ui be a projection map from t/(g) to the symmetric algebra 5(a) 
of ac = a <S)r C along the decomposition 

C/( ) = 5(a) (nC/( ) + U(g)n). 

It is known that uj is an algebra homomorphism from Z(g) into 5(a). We can 
identify the symmetric algebra 5(a) with the algebra of polynomials on a^. 
Hence if we consider the evaluation of lu(-) G 5(a) at cZAe, we obtain a character 
of Z(q) as follows 

XX- Z{q) 3 X .— > Lo(X)(dX e ) G C. 
We define a subspace of C°°(G/ K) by 

G°°(G/X;.M A ) = {/ G C°°(G/K) \ R x f = Xx (X)f for X G Z(fl)}. 
We put 

The following theorem is known as Helgason's conjecture [5]. 
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Theorem 2.4 (|14j). The Poisson transform V x gives G '-isomorphism 

B(G/P ;\e)=C°°(G/K;Mx) 
if and only i/e(Ae) 7^ . 

We can also define the Poisson transform for the subspace B(G/Pq;X) of 
B(G/P ; Ae). We discuss the characterization of the image of B(G/Pq; A). We 
consider the subspace 

C°°(G/K; J e (A)) = {/ e C°°(G/K) \ R x f = for X e I@(X)} 

of C°°{G/K-M X ). 

Remark 2.5. We can easily show that 

Ie(A)D ]T U(q)(D-u(D)(\ @ )) 
Dez( s ) 

(cf. Remark 4-3 in '221). Hence actually C°°(G/K;I®(\)) is a subspace of 
C°°(G/K;Mx). 

We assume 

Ae + p G 0^- is regular and dominant. 
Here p = |tr(ad|„) € a£, i.e., 

1 " n + 1 

l<z<j<n i=l 

This assumption is equivalent to 

$ {0 , -l, - 2) . • ■} for a e A+( flj a), 

i.e., 

(A^ +«,-)- (Aj + I/*) £ {0,-1,-2,.-.} 

for i < j and are integers which satisfy nk-i + 1 < Vk < (fc = i or j). We 
keep this assumption all through the remaining of this paper. 

Theorem 2.6 (Oshima. Theorem 5.1 in |22j). Under the above assumption, 
the Poisson tranform 

T*:B(G/P e ;\) — C°°(G/K,I e (\)) 

f 1 — > = / f(xk)dk, xeG. 

is a G -isomorphism. 
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2.3 The explicit generators of Je(A). 



In the previous section, we see that a degenerate principal series representa- 
tion has a realization on the subspace of C°°(G/K) which is the kernel of the 
annihilator ideal ie(A). In [19], [20] and [22], T.Oshima obtained several good 
generator systems of We introduce one of his generators here. 

We denote the space of n x n matrices with entries in U(g) by M(n; U(g)). 
For E = (Eij)ij E M(n; U(g)), we define elements in Z(g) by 

A,-=tr(E fc ), for k = l,...,n. 

Then it is known that Z(g) = C[Ai, . . . , AJ as C-algebras. 

Theorem 2.7 (Oshima. Corollary 4.6 in [22])- Asuume Ae + p € is regular 
and dominant. Then we have 

n n L L—l 

J e(A) = E u <&) 11^ - Afe - + E ^(fl)( A * - XA(A fe )). 

i=l j = l fc=l fe=l 



3 Generalized Whittaker models 

The generalized Whittaker model is the main theme of this paper. We will 
give a characterization of the space of the generalized Whittaker models of a 
degenerate principal series 7re,A as the kernel of This is an analogy of 

Yamashita's method in the case of irreducible highest weight modules [33] . The 
substantial part of his method is that the maximal globalization (in the sense 
of W.Schmid [25]) of highest weight modules is given by the kernel of a certain 
differential operator. The corresponding theorem for the degenerate principal 
series is obtained in Theorem 12.61 in Section 12.21 Moreover thanks to Theorem 
12.71 we know explicit structures of these differential operators. Hence we can 
carry out the explicit calculations about the space of the generalized Whittaker 
models. 

Let Vk be the space of fsT-finite vectors for a continuous representation of G 
on a complete Hausdorff locally convex space V . Let Xq^\ be C°° (G / P®\ A) if. 
This becomes a (gc, iQ-module, i.e., the gc-action is the differentiation of Tro^ 
and the if -action is the restriction of 7re,Aj furthermore the actions of gc and K 
are compatible. Also Xq^\ is a Harish-Chandra module, i.e., finitely generated 
as a [/(g)-module and with finite if- multiplicities. 



3.1 Maximal globalization 

For the Harish-Chandara module X@,\, let us consider its dual Harish-Chandra 
module Xo,a* ■ Here the character A* of P® is defined by 

A* = -A - 2,00 = (n - no — n\ - A, . . . , n - n L -i - n L - A), 

where pe = ^tr(ad| ne ) G a£, i.e., 

L 



EBj-i + rii — n v-^ 
2 E 

i=i j=m-i 
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Actually, if we consider the pairing ( , ) : C°°(G/P e ; A) x G°°(G/P e ; A*) -> C 
denned by 

(f,g) = / f(k)W*)dk 

JK 

for (/,<?) € C°° (G/ Pq; X)xC°° (G / Pq; X*), this is a G-equivariant non-degenerate 
sesquilinear pairing. By this pairing, the Harish-Chandra module A"e.A* = 
C°° (GJ Pq.\* )k can be identified with the dual Harish-Chandra module (Xo,a)* ,i-e., 
all if -finite vectors in Homc(Xe,A, C)k- Here if acts on Homc(Ae,A,C) by 
k ■ I(v) = Iiire^ik-^v) for I G Homc(X e> A, C) K and v G X e , X - 

We can consider the natural (qc x cj c , if x if )-bimodule structures on A"e,A <8 
l e ,A> and C°°(G). For Xl,X 2 G flc and fci,fc 2 G if, we put 

(X l5 X 2 )(/ <g> /*) =7r ei A. ® /* + / ® ^e,A* (X 2 )/*, 

(fci, fc 2 )(/ ® /*) =ne,\(ki)f ® Tre.A. (**)/' 

for / G Xe.A and /* G Xq,\*. Also we define 

(Xi, X 2 )g =L Xl g + Rx 2 g, 
(k 1 ,k 2 )g =L kl R k2 g 

for g G C°°(G). Then we introduce the matrix coefficient map (cf. [4]) from 
(flc x 0c, K x if )-bimodule X e ,A ® Xq.\* to G°°(G) so that, 

1. the map c: A e ,A <8 A e ,A* — > C°°(G) is a (gc x 0c,^ x if)-bimodule 
homomorphism , 

2. for any / G Ae,A and /* G Ae.A*, the evaluation at the origin e G G 
becomes 

c(/®/*)(e) = {/,/*)• 

It is known that this matrix coefficient map is uniquely determined (cf. Theorem 
8.7 in H). 

If we consider the restriction of Poisson transform Vq on Ao,a, Theorem l2.6l 
gives us the (gc, if )-isomorphism 

Vb- X e ,x ^C°°(G/K;I e (\)) K . 

Lemma 3.1. Take the K-fixed vector fo G Ae,A* such that /o|k = 1- Then 
the restriction of the Poisson transform on Xq \ is a matrix coefficient of an 
element of X@ t \ with /o G Ao,a*, *-e-, 

^(/) = c(/®/ ). 

Proof. By the pairing of C°°(G/Pq;\) x G°°(G/P e ; A*) defined above, we can 
define a map le,A ® ^e,A« — * G°°(G) as follows, 

/®/* — > (^.a^ 1 )/,/*) = / f{gk)TW)dk, 

JK 
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for / G X@ } \ and /* G Xe,A*- This map satisfies the conditions of the matrix 
coefficient map. Hence for / G Xe,A, we have 

H(fM= I f(9k)dk 

JK 

= I f(gk)Mk)dk 

JK 

= I {■Ke,x{g- 1 )f){k)MK)dk 

JK 

= (ne.Ag- 1 ).f,.f ) 
= c(/®/o)( 5 ), 

by the uniqueness of the matrix coefficient map. □ 

Let us consider the space of (gc, if)-homomorphisms of Xe,A> into C°°(G), 

Kom gCtK (X @ , x ,,C°°(G)). 

Here we regard G°°(G) as a (qc, -fO-module by the right translation. More- 
over, this space of (gc, if )-homomorphisms inherits a Frechet topology and 
a continuous G-action from G°°(G). More precisely, we define a semi-norm 
on this space as follows. The space G°°(G) is a Frechet space of uniformly 
convergence on compact sets for functions on G and their derivatives. Let 
{| • | q }q£a be a family of countable many semi-norms on C°°(G) which defines 
the Frechet topology on G°°(G) where A is the index set. Take a semi-norm 
I • U € {| • | q } q gA a nd v G le,A*- Then we define a real-valued function 
| • \ a , v : Kom gCiK (X @ ,x*, C°°(G)) ^ K> by 

\I\a,v = \I(v)\ a 

for I G Hom flCj if(Xe,A, , G°°(G)). We can see that the function | • \ aiV defines a 
semi-norm on Hom BCi ^(Xe,A, ; G°°(G)) for a £ A and v G Xq : \*. 

Lemma 3.2. Let {v n } be a countable vector space basis of the Harish- Chandra 
module Xe,A« • Then the family of semi-norms {\ ■ \ a ,v m } a eA,v m e{v n } defines a 
Frechet topology on Hom gCi ^ (Xe,A» , G°°(G)). 

Proof. Take I G Hom Bc> x(^e, a*, G°°(G)), and we assume |/| ail , m = for any 
I ' \a,v m G {| • \a,v m } a eA,v m e{v n }- Since {v n } is a basis of X e ,\*, it follows 
that = for any v G X<~,^\, and a G A. This means I(v) = for 

any v G A^a* because G°°(G) is the Hausdorff space. Thus we have I = 
0. This implies that Hom flCi x(^e, a», G°°(G)) is a Hausdorff space as well. 
Since {| • \ a ,v m } a eA,v m e{v n } consists of countable many semi-norms, the space 
Hom gCi ^(A'0 i A», G°°(G)) is metrizable by this family of semi-norms. Finally we 
need to check the completeness. Suppose that there exists a Cauchy sequence 
{Ik} of the elements of Hom BCi ^(Xe, a, , C°° (£?)), i.e., 

\h-h\ a,v m — * for fc, I — > CO, 

for any | • \ a .v m G {| • \a,v m } a eA,v m e{v n }- Then this implies that for any v G 
Ao,a«, the sequence {Ik(v)} G G°°(G) is a Cauchy sequence. Hence there 
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exists limfc-,00 J fc (u) € G°°(G). We define the map f: X e ,A* -► C*°°(G) by 
7(u) = linifc^oo Ik(v) for u G Xe,A* ■ Then we will show that / is the element in 
Hom flCi ^(Xe,A, , C°°(G)). For any Z £ qc, v £ Xq : \* and a 6 A, we have 

\I(veMZ)v) -R z I(v)\ a 
= |J(7r e ,A- (^)«) - 4(7re,A* (2)«) + Rzh(v) - R z I{v)\ a 
< \I(ir e ,x*{Z)v) - I k (ire,x*(Z)v)\ a + \R z I k (v) - R z I{v)\ a 
— > for k — » oo. 

Thus I(7Te,A* (^!M = R Z I{v) for any Z 6 gc and i> € ^e,A*- Hence 7 is a 
gc-homomorphism. Similarly we can show that 7 is a TT-homomorphism and 
a linear map. Hence we could show that 7 € Hom gc t ic{X®,\) C°°(G)). By the 
construction of 7, we can see that Ik —> I (k —> oo) in Hom BCi s-(Je,A« , G°°(G)). 
This proves the lemma. □ 

We could define a Frechet topology on Hom Sc ^(Je,A, , C°°(G)). Then a 
continuous G-action on this space is defined by left translation on G°°(G). Hence 
this space defines a continuous Frechet representation of G. This is called the 
maximal globalization of the Harish-Chandra module Xq^\ (cf. and [T5]). 

Lemma 3.3. We assume that Xe,A* is irreducible. Take the K-fixed vector 
fa £ Ye, a* such that /q|k = 1. We consider a mapping 

$: Hom 8 ^(l e ,A., C°°(G)) — > G°°(G) 

/ — > /(/o)(ff) (jeG). 

TTien $ is a continuous mapping. Moreover for any semi-norm \ ■ \ a ,v m £ {| • 
U.f m }aeA,« m e{u„} ok Hom gc .x(Xe.A* : G°°(G)), i/iere exists a continuous semi- 
norm fi a .v m on G°°(G) such that 

fJ*a,v m ($(I)) = \I\a,v m , 

for I e Hom Bc , K (X e ,A*, G°°(G)). 

Proof. For any semi-norm | • | Q on G°°(G), there exists a continuous semi-norm 
| • | a ,/ on Hom 8C! x(^e,A* > G°°(G)) such that 

= l^(/o)|a = 

for 7 € Hom BC! jf(Xe, a*, G°°(G)). Hence $ is the continuous. Conversely, we 
take a semi-norm | • \ atVm on Hom 8c ^(Xe,A* , C°°(G)) for a € A and v rn £ {v n }. 
Since Ye, a* is the irreducible Harish-Chandra module, there exists an element 
X £ 1/(2) such that 7Te,A*(^0/o = v m- Then we have 

\I\ a ,v m = \I(v m )\ a = \I(n @ ,x*(X)f )\ a = \R x I(f )\ a . 

If we recall that U (g) can be identified with the ring of left invariant differential 
operators on G°°(G), then Ma,i/ m (/) = \Rxf\a defines a continuous semi-norm 
on G°°(G). This proves the lemma. □ 

The following proposition is essentially obtained in [26] and [15] (more direct 
proof is also given by H. Yamashita [53]). We give a proof for the completeness 
of the paper. 
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Proposition 3.4. We assume that Xo,a* *s irreducible. Take the K-fixed vec- 
tor fo G Xq^x* such that /o\k = 1- Then we have a following topological 
G -isomorphism, 

$: Hom BC , K (X e ,A*, G°°(G)) C°°(G/K; /©(A)) 

* /(/o)(s) (jeG). 

-ffere C°°(G/K; Iq(X)) has the Frechet topology as the closed subspace of C°° (G) . 

Proof. We can immediately see that $ preserves the action of G by defini- 
tion. First we show that $ is well-defined. Take a if -finite element I S 
Hom gCi jf(Je,A'i C°°(G))k- Then by the evaluation at the origin e G G, we 
can regard I(-)(e) as the element of X e ,A — (ATe.A*)*- Since I(fn)(g) G G°°(G) 
is if-finite and Z(g)-finite, it is a real analytic function on G. Let VF be an suffi- 
ciently small open neighbourhood of in g. Then we have the Taylor expansion 
at the origin e G G, 

/(/o)(expX) = £-fl x „(/(/o))(e) 

= E^( / (')(e), 7 re,A*(X")/„> 

= E^! c ( / (-)( e )® 7r e,A.(X")/ )( e ) 

= E^*" c ( J 0(e)®/o)(e) 

= c(/(-)(e)®/ )(expX) 

for X G W. We can extend this equality for the identity component G° of 
G because both functions are real analytic. And the fact G = G° ■ K implies 
I(fo)(g) = c(7(-)(e) ® fo)(g) for all g £ G. Hence by Theorem 12.61 and Lemma 
13.11 we have the inclusion 

$(Kom BC! K(Xe,\* , C°° (G))k) C C°°(G/K; 7 (A)). 

We recall that for a continuous representation of G on a locally convex complete 
space V, the space of i-T-finite vectors Vk is dense in V (for example, Lemma 
1.9, Ch.IV in [5]). Since $ is a continuous mapping by Lemma l3~3l we have 

$(Hom JC , K (X 9 ,A., G°°(G))) = $(Cl(Hom Bc ^(X e> A.,G 00 (G)k) 
C Cl($(Hom 8c ^(X ,A.,G oo (G))K)) C Cl(G°°(G/K;/e(A))) 
= C°°(G/K;I e (\)). 

Here Cl(-) is the closure. Hence <£> is well-defined. Next we prove $ is the 
bijective map. Since Xe,A* is irreducible, the map $ is injective. We will prove 
that $ is surjective. 

For any F(g) G C°°(G/K; I&(X)) K , there exist h G Xq,a and F(g) = c(h(g> 
fo)(g) (.9 S G) by Theorem 12.61 and Lemma 13.11 We define an element of 
Kom gCtK (X e , x , G°°(G)) so that 

//,(«) (ff) = c(h®v){g), 
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fow G X e ,A*- Then we can see that <$>(h)(g) = 4(/o)0) = c(h®f )(g) = F(g). 
Hence we have an inclusion C°°{G/K;I e {X)) K C $(Hom Bc>K (Xe,A, C°°(G))). 
Because C°°{G/K]Iq{X))k is a dense subspace of C°°{G/K\I@{\)), for any 
/ € C°°{G/K\I®{\)) we can choose a convergent sequence /„ — > / (n — » oo) 
where /„ G C°°{G/K;I®{\))k for n G N. The above inclusion shows that 
there exist 7„ G Hom 9£i x(^e, a, C°°(G)) such that = /„. From the 

second assertion in Lemma 13.31 the sequence {/„} is a Cauchy sequence in 
Hom flCj #-(Xe,A, C°°(G)). Since Hom flCi ^(Xe,A, C°°(G)) is a Frechet space, i.e., 
complete space, there exist I G Hom 0Ci if (Xe^, C°°(G)) such that I n —> I 
(n — > oo). Thus we have $(7) = (/) by the continuity of <f>. This shows 
that <i> is a surjective map. The open mapping theorem leads that $ is a 
homcomorphism. □ 

3.2 Generalized Whittaker models 

We define a generalized Whittaker model for Xq^\. Let us fix a closed subgroup 
U of N. We take an irreducible unitary representation r\ of U on a Hilbert 
space V v . Let V£° be the space of C°° -vectors in V n . Let us consider the space 
C™(U\G) = {/: G -> smooth |/> 5 ) = r)(n)f(g),9 € G,n G [/}. This 
becomes a G-module by the right translation. 

Definition 3.5. We consider the following intertwining space 

Kom Sc , K (Xe,\-,C™(U\G)). 

We call images o/Xe,A* by these (qc, K)-homomorphisms generalized Whittaker 
models o/ Je,A* • 

Theorem 3.6. Assume that Xq^\* is irreducible. We take the K-fixed vector 
in Xe,A* such that fo\n = 1- Then the following mapping 

Kom BCiK (X @ ,x*,C™(U\G)) ^ C™{U\G/K; 7 e (A)) 
VF .— > W(/ )(<7) 

is a linear isomorphism. Here 

C™(U\G/K;I @ (\)) 

= {f:G^V™ smooth \ f(ngk) = 7?(n)/(g), .g G G, n G 17, fc G K 

andR x f(g) = 0, X e I e (X)}. 
Proof. Fix a nonzero element £ G V n . Then we consider the linear mapping 
T: C?(U\G) G / i > (£,/(<?)>„ G C°°(G), 

which commutes with G and flc actions from the right where ( , } v is an inner 
product on V v . Since (77, V^) is an irreducible unitary representation of U, this 
mapping T is injective. In fact, if T(f) = for / G C™(U\G), then we have 

= T(f)(ng) = (£, f(ng)) = (£, n(n)f(g)) = M*" 1 )*, f(g)), 

for any n G U and g £ G. Since is irreducible, this implies / = 0. By this 
map T, we also have an injective map 

f : Hom (BC>K) (X e ,x*,C™(U\G)) — » Hom (BCiJr) (X e , A , , C°° (G)) 
VF 1 — ► Toff. 
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For any W £ Hom (BC , K) (Xe, A ,, C°°{U\G, r?)), we have T($(W)) = T{W(f Q )) = 
ToW{fo) = f(W)(f ) = $(T(W)). Hence we have the following commutative 
diagram, 

Hom (gC)K) (X e , A ,,C-(C/\G)) — ?— C™(U\G/K) 

T 

Hom (gc>JC) (Xe, A .,C 00 (G)) G°°(G/K) 

Since 3>, T and T are injective, we see $ is injective. Next we show that 
Im$ c C™(U\G/K;I @ (X)). Take W £ Kom (gCtK) (X @ ,x*,C™(U\G)), then 

T($(W))(jg} = ((,f(/o)(5)) e C°°{G/K;I e {X)) where g € G. Hence we have 
= = T(R x $(W))(ff) for X e J e (A) and g e G. Since T 

is injective, we have RxW(fo){g) — for X £ Ie(X) and g £ G, i.e, Im$ C 
C^(U\G/K;I e (X)). 

Finally, we show that $ is surjective. Let / G C™(U\G/K;I e (X)). For 
w € -^e,A* there exist X v £ U(q) such that v — ne,\*(X v )fo since le,A* 
is irreducible. Then we define a mapping Wf. Xe,A* 3 v = TtQ,\*{X v )fo l— * 
R-Xvfig) G G^°([/\G). We need to check that it is a well-defined mapping. 
If for X„,X[, e § we have v = 7r .A* (X v )f — w@ t \* (X' v )fo, then we have 
ne,\*{X v - X' v )f = 0. On the other hand we have T(f) £ C°° {G/ K; I e {X)). 
Thus there exists 7/ <E Hom 8CjK (l e ,v , G°°(G)) such that T(/) = $(//) by 
Proposition EU We put Z = X„ - X' v . Then we have T(R z f) = R z T(f) = 
$(R z If) = R Z If{fo){g) = If(ne,\.(Z)fo)(g) = 0. Hence by the injectivity of 
T, we have Rzf(g) — 0, i.e., Rx v f = Rx< v f- This implies that W/ is well- 
defined. Also we can check that Wf is compatible with gc and K actions. 
Hence W f £ Hom (8c>K ) (JTe.A* , C™(U\G)) and = W>(/ ) = /. Hence $ 

is surjective. □ 

Remark 3.7. This theorem is an analogue of Yamashita's result for the gen- 
eralized Whittaker models of discrete series representations (Theroem 2.4- in 
]/J and more general settings (Corollary 1.8. in \3ty). 



4 Calculus on the case of GL(4, R) 

In previous sections, we gave a characterization of the space of generalized Whit- 
taker model as the kernel of some explicit differential operators. We will cal- 
culate some examples on GL(4, R) by using these theories. In particular we 
take the spherical degenerate principal series representations induced from the 
maximal parabolic subgroups Pi,4, -P2.4 and compute dimensions of the spaces 
of generalized Whittaker models and find the basis for them. 

Let us explain the detailed settings. We consider the case n = 4. Hence 
G = GL(4, R), K — 0(A) , A is the group of the 4x4 diagonal matrices with 
positive real entries and N is the group of 4 x 4 strict lower triangular matrices 
with Is in the diagonal entries. We put Pk = Pk,A, k = 1,2. For (Ai, A2) £ C 2 , 
we define the character X: Pk — > C x and define degenerate principal series 
representation induced from Pk and A as before. Let Xk t \ be their Harish- 
Chandra modules which consist of X-finite vectors of these degenerate principal 
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series representations. Then by Theorem 12.71 the annihilator ideal in U(g) of 
the degenerate principal series representation are written by 

4(A)=/ {M} (A) 

4 4 4 

J2 U ^ E - Al )( E - A 2 - fc) + Ufa)Q2 E a - A i - ( 4 - fc ) A 2). (4.1) 

i— 1 j—1 i—1 

for fc = 1,2. We put a stronger condition for A, 

Ai - A 2 <^ Z. 

4.1 Equivalent classes of C™(U\G). 

A generalized Whittaker model is an image of an embeddings of Xq \<- into 
C^(U\G) where U is a closed subgroup of N and r\ is its irreducible unitary 
representation. In this paper, we consider the space C™(U\G) defined as fol- 
lows. 

1. the group U is a closed subgroup of N and r\ is its unitary character, 

2. the unitary induced representation L 2 -Ind^f r/ is an irreducible unitary 
representation of N. 

We will classify the G-equivalent classes of these C™(U\G). 
4.1.1 The classification of N 

Firstly we will give the classification of the unitary dual of maximal unipotent 
subgroup N of G by using the Killirov's method for coadjoint orbits. This 
section contains no new results. The details of the contents of this section can 
be found in [3] for example. 
Let n = Lie(iV), i.e., 



n(z,yi,y 2 ,xi,X2,x 3 ) 



/0 

x 1 

Vi x 2 

\z y 2 x 3 0/ 



We denote its dual M-vector space by n* = Honit(n,R). We identify this space 
with a subspace of M(4, R), 



l(0i, 01,02, 71, 72,73) 



fO 7i 01 a\ 

72 02 

73 

V 0/ 



,73 G 



so that 



l(a,0i, 02, 7i> 72, 73) ' n(z,y 1 ,y 2 ,x 1 ,x 2 ,x 3 ) 

= tr(/(a, 0i,02, li,l 2 , l 3 )n(z, yi,y 2 , Xi,x 2 ,x 3 )) 

= az + 0iyi + 2 y 2 + j x xi + i 2 x 2 + i 3 x 3 . 
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We define the coadjoint action of N on n by (Ad* (n)l)(X) = Z(Ad(n. _1 )X) for 
n G N, I 6 n*, X 6 n. Take a basis X\,X2, X 3 , Y\,Y 2 ,Z of n and its dual basis 
X{ , XI , A 3 * , Y{ , Y 2 * , Z* of n* so that 

2 3 

n(z,y 1 ,y 2 ,x 1 ,X2,x 3 ) = zZ + '^y i Y i + '^x j X j 

i=l j=i 

and 

2 3 

l(a, 0i , 02 , 7i , 72 , 73) = aZ* + £ ftF/ + £ 7,- A* . 

i=i 3=1 

Under our coordinate system, the coadjoint action is written as follows, 

(Ad* exp(n(z, ■ ■ ■ ,x 3 )))(l(a, ■ ■ ■ ,73)) 
= aZ* + (/?i + ax 3 )Y* + (f3 2 - ax x )Y 2 * 

+ (71 + 0iX2 + a(y 2 + ^p-))Xf + (72 + x 3 {3 2 - xxPx - Xl x 3 a)X* 2 ( 4 - 2 ) 

+ (is-^ + a(Y-yiM. 

We consider the classification of coadjoint orbits of n*. First, we assume 
that a 7^ 0. Then by the equation (|4.2|) , if we choose appropriate £3, sci, 2/2, J/i, 
we can find in the Ad*A-orbit a point with 0\ = 2 = 7i = 73 = 0. Hence if 
we write l a .j 2 = ^( a i 0, 0, 0, 72, 0), the coadjoint orbit is written as follows, 

(Ad* N)l an2 = {aZ* + tiY{ + t 2 Y* + siX* 

+ (j2 + —)X;+s 2 X; ■ t 1: t 2 , Sl ,s 2 eM.}. 
a 

Next, we consider the case a = 0, i.e., 1(0, 0\, 02, 71, 72, 73)- We assume 0i ^ 
or ft / 0. Then from the equation (|4.2j) . we can see that there is an element 
1(0,01, /?2,7i,0, 73) in (Ad*iV)Z(0, ^i,/3 2 ,7i,72,73)- Hence in this case, it is 
enough to consider the orbit 

(Ad*N)l(O,0 1 ,0 2ni ,O, l3 ) 

= {0xY* + 2 Y 2 * + (fati + 71) A* 

+ t 2 X* 2 + (73 - 2 ti)X; ; h,t 2 G R}. 

Finally we consider the case 0\ = 02 = 0. Then we have 

(Ad*n*(0, 0,0, 71,72,73)) = {71 Ar +72A* + 73 A 3 *}. 

We summarize these as a proposition. 

Proposition 4.1. We can classify coadjoint orbits of n* in following cases. 
(I) For a G R\{0} and 72 G R 7 

O a , 72 =(4^ A)Z(a, 0,0, 0,72,0) 

={aZ* + hY* + t 2 Y 2 * + Sl Xt + (72 + — )A 2 * 

a 

+ s 2 X* ; h,t 2 ,Sx,S2 G M}. 
JFe have dimO a ^. 2 —4- 
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(II) For fa, /3 2 ,7i, 73 G K swc/i tfiarf /3i/3 2 7^ 0, 



Oft ,02,71.73 

={PiY* + p 2 Y* + {piti + 7l )X x * + < 2 X 2 + ( 73 - /3 2 ti)X 3 * ; 

ti,t 2 e M}. 

(Ill) For 71,72,73 G R, 

^ 71 ,72,73 =(^A0'(0, 0, 0, 71, 72, 73) 

={7l*l* +72*2* +73*3*}. 

We /lave dim C 7l i72 j73 = 0. 

To construct the irreducible unitary representation of iV from the coadjoint 
orbit of I e n*, we should determine its radical ti and maximal subordinate 
subalgebra S/. We define the coadjoint action of the Lie algebra nonl e n* by 
((ad*X)/)(Y) = l([Y,X}) for X,Y e n. 

Definition 4.2. For I S n*, we define the subalgebra of n smc/i f/iai 

r,={Ien; (ad* X)/ = 0}. 

We caZZ £/ws subalgebra the radical of I e n* . 

If y is a R-vector space with an alternating bilinear form B, its isotropic 
subspace W is the subspace such that B(w, w') = for all w, w' e W. We define 
the radical of -B by radi? = {v E V ; B{v, w) — 0, all w € V}. It is known that 
any maximal isotropic subspaces of V have codimention idiniR^/radi?). 

Definition 4.3. For I S n*, we can regard l([X,Y]) as a bilinear form for 
(X, Y) e nxn. By the antisymmetry of Lie bracket [X, Y] — — [Y, X] (X, Fen), 
this is an alternating form on n x n. T/ie subalgebra S; C n which is isotropic 
fori and has codimension ^dim^n/vi) is called maximal subordinate subalgebra 
of n for I. 

Remark 4.4. For a nilpotent Lie algebra n, there exists at least one maximal 
subordinate subalgebra for any Zen*. Although the radical for I is uniguely 
determined, maximal subordinate subalgebras are not unique for I. 

Let us construct radicals and maximal subordinate subalgebras for coadjoint 
orbits (I), (II), (III) which are classified in Proposition 14. II 

The case (I). By the equation (14. 2p . the coadjoint action of N on l a , l2 = 
l(a, 0, 0, 0, 72, 0) is written as follows, 

(Ad* exp(n(z, • • • , x 3 )))l(a, 0, 0, 0, 72, 0) 

= {aZ* + yiY* + y 2 Y* + Xl X{ + (72 + —)X* + x 2 X* 3 }. 

a 

Hence we can see that 

r iaT2 = {RZ + RX 2 }. 
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As we noted before, there are some choices of maximal subordinate subalgebra 
even if it contains the radical tj a . Among these choices, we pick up a maximal 
subordinate subalgebra 

si an2 = {RX 2 + RYi + RY 2 + RZ}. 

It is easy to check that this subspace is isotropic and its codimension is equal 
to idimn(n/ti Q 72 ) = 2. Also this becomes a subalgebra of n. We can see that 
Si a 72 does not depend on the choice of a £ K\{0} and 72 £ M. Hence we simply 
write s (/) = s ia ^ . 

The case (II). As well as the case (I), we can see that the radical for 
hufartui* = K Q ,01> fo> 7i) 0)73) is given by 

V ,^,,3 = {R(PiX 3 + faX{) + RYi + RY 2 + RZ}. 

The codimension of maximal subordinate subalgebras are ^dim^n/t^ 02 7l ) = 
1. We recall a fact for the codimension 1 subalgebra of n. 

Proposition 4.5 (cf. Proposition 1.3.4 in [3]). Let g be a nil-potent Lie algebra 
and go a codimension 1 subalgebra ofg. Fori £ g* , letlo — l\ So be the restriction 
to Qq. If the radical of I in g is contained in go, any maximal subordinate 
subalgebra of go for Iq is also maximal subordinate subalgebra of g for I. 

For any codimension 1 subalgebra no of n containing ti Pl 02 yi 73 , there exist 
a maximal subordinate subalgebra of no for '/3i,/3 2 ,7i,7 3 |n from Remark 14.41 By 
this proposition, this is also a maximal subordinate subalgebra of n for l( U y By 
the calculation done above, this maximal subordinate subalgebra should have 
codimension 1. Hence this is nothing but no. This implies that any codimension 
1 subalgebra no of n containing ^l Plt p 2nin3 is maximal subordinate subalgebra 
for ^/3i,/3 2 ,7i,73 ■ Among these, we pick up a maximal subordinate subalgebra 

^ft^m/ra = i RX ^ + RX * + KY 1 + MY 2 + RZ }' 

As in the case (I), the subalgebra Si ffl ^ 73 does not depends on the choice of 
Pufe, 7i, 73 £ K- Hence we simply write S( U) = Si^ ^ ^^ . 

The case (III). The coadjoint action on Z 71 72 73 = Z(0, 0, 0, 71, 72, 73) is given 

by 

(Ad* exp(n(z, • • • , x 3 )))l(0, 0, 0, 71, 72, 7s) = {71*1* + 7 2 A 2 * + 73^3 }■ 
It is obvious that the radical of J-^ ,72,73 is 

ti = n. 

l 71.72'73 

Also it is obvious that a maximal subordinate subalgebra of 71,72,73 is 

S 7l ,72,73 = n - 

Let us recall Kirillov's theory for irreducible unitary representations of nilpo- 
tent Lie group N. For I £ n* , let S; be a maximal subordinate subalgebra for I 
and let Si — exps; . We can extend l\ Sl : S; — > R to the map Xl '■ &l ~* by 

X; (expX) = e 2rf W, X£ 5l . 
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This is a group homomorphism, i.e., an unitary character of Si because S; is an 
isotropic subspace for I, i.e., l([X,Y\) = for X, Y G s;. We consider a Hilbcrt 
space induced from xu 

H Xl ={/: N — > C measurable ; /(sx) = Xl{ s )fi x ) f° r s £ Si,x € N, 




where is the right-invariant measure on Si\N. The inner product is defined 

by 

(/,/') = / f{x)J{x)dx. 

JSi\N 

It can be shown that TL Xl is complete by this inner product. The action of N 
on H. Xl is given by the right translation. This is the unitary representation by 
the right-invariance of dx. This representation is called the representation of N 
induced from Xh an d denoted by L 2 Tnd^(xz). 

Theorem 4.6 (Kirillov [13 ). Take I G n* and let S; be a maximal subordinate 
subalgebra of n for I* . 

1. The induced representation L 2 -Ind^(xz) is an irreducible representation 
of N. 

2. Let s\ be a maximal subordinate subalgebra of n for I and S^ = exp sj . 
Then L 2 -Ind^/ (xi) is unitarily equivalent to L 2 -Ind^(x;). Hence we may 
write tti for £ 2 -Ind^(xz)- 

3. Let V G n* . Then ixy is unitarily equivalent to tti if and only if V G 
(Ad*N)l. 

4- Let 7r be an irreducible unitary representation of N. Then there exists an 
I G n* such that 7r is unitarily equivalent to 7T; . 

This Kirillov's theorem implies that irreducible unitary representations of 
nilpotent subgroup N of G are equivalent to induced representations Ind^(xz) 
and their equivalent classes only depend on coadjoint orbits of I G n*. We 
have already classified coadjoint orbits of I G n* and determined their maximal 
subordinate subalgebras S; . Hence we can obtain equivalent classes of irreducible 
unitary representations of N. 

Proposition 4.7. We retain the notation as above. The every irreducible uni- 
tary representation of N is unitarily equivalent to one of the following represen- 
tations. 

(I) For l a ,-y 2 = K a > 0, 0, 0, 72, 0) G n* and its maximal subordinate subalgebra 
S(i) = + KYi + RY2 + RZ}, we define the representation 

L 2 -Ind" xi ■ 

Here Sm = expsm and a G R\{0},72 G R. 
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(II) For 1^,02,11,13 — ^(0, /?2, 7i, 0, 73) € n* and its maximal subordinate 
subalgebra s (II ) = {R^i + RX 3 + RYi + RY 2 + RZ}, we define the repre- 
sentation 

Here Si (n) = exps/ fw; and ft , ft , 71 , 73 G R, ft ft 7^ 0. 

(777) For Z 7l 72 73 = Z(0, 0,0,71,72,73) G n*, we define the unitary character of 
N, 

,72 .73 ' 

4.1.2 Conjugacy classes of C™(U\G) 

In previous section, we classify the unitary dual of N. Thus the next thing to 
do is the classification of G-equivalent classes of the following spaces, 



C " Q „ 2 (%A G )' " G R\{0}, 72 G M, (I) 
G~ (%/)\G), ft,ftGR, ftft 7^0, (II) 

^,^,,3^)' 71,72,73 GR. (Ill) 

For x e G, we write the conjugation of g e G by x as g x = xgx -1 . Let H be 
a closed subgroup of G and 7r a continuous representation of if on a complete lo- 
cally convex space E. Then for x e N G (H) = {g e G\h 9 e 77, for any ft € 77}, 
we can define conjugation of tt as n x (h) — Tr(h x ). Then we have the follow- 
ing fact about the induced representation G£°(77\G) = {/: G — ► F smooth | 
/(%) = n(h)f(g), g € G,h e 77} on which G acts by the right translation. 

Lemma 4.8. We retain the notations as above. The map 

C?(H\G) ^ C™{H\G) 
/(<?) — > F(g) = f(xg) 

gives isomorphism as G-modules. 

Proof. We see that this map is well-defined. Take / e G£°(77\G) and define 
F(g) — f(xg). Then we have 

F(hg) = f(xhg) 

= f(xhx~ 1 xg) 
= ■nixhx^fixg) 
= ir x (h)F(g), 

for h G H. Hence F G G£? (H\G). Obviously this map is bijective and preserves 
the action of G. □ 

Lemma 4.9. Fix a maximal subordinate subalgebra S; C n for I € n*. We 
put Si = exps;. Let us define a character Xi : $1 ~ * C 1 so that %/(expX) = 
e 27r\/^T;(jf ) y or £ S; T/ien ifce character xi * s invariant by the conjugation 
by Si, i.e., 

Xt(s)=Xl(s) 

for s,x E Si. 
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Proof. Take an element of x € Si. Then there exists Z x e S; such that x = 
expZ x . By the Campbcll-Hausdorff formula, for any X e Si we have 

i(expX)i _1 = expZ x expXexp(— Z x ) 
= cxp(Z x + X - Z x + Y) 
= exp(X + Y) 

for some element Y e [s;,S;] = {[V, VF] | V, W G Sj}. If we recall that S; is a 
maximal subordinate subalgebra for I, we have = for Y e [s;,Sj]. Thus 
we have 

X?(expX) = X i{x(^V X)x- 1 ) 
= Xi(cxpX). 

for any X E Si. 

By these lemmas, we have the following classifications. 
Proposition 4.10. C'ase(I). For a e E\{0} and 72 <E E, we /mue 



X '- J %A ^"l%.o.,o.o.o)(^\G) ^/7 2 = (/ 2 ) 

Case(II). We take (3\, /3 2 , 71, 73 € E one? assume 7^ 0. TTien we /icrae 



□ 



C. 



Xi 



01, /3 2 , 71, 73 



Xi(o, 0,1, 1,0,0) 



(S f /7)\G) */(/3i,7i)-(73,/3 2 )^0 

^Xi(o, 0,0, 1,0,1) 



W,7i)-(73,/3 2 ) = 
and /3i ^ 0, [3 2 ^ 



^Xi(o, 0,0, 1,0,0) 



^Xi(o, 0,0, 0,0,1) 

(S ( ii)\G) 



*/(/3i,7i)-(73,/3 2 ) = 
and ft ^ 0, /3 2 = 

*/(/3i,7i)'(73,/3 2 ) = 
and /3i = 0, #2 7^ 



(Hi) 
(Ih) 

(Ih) 

(Ih) 



where (a,b) ■ (c, d) = ac + bd for a,b,c,d G E is a natural inner product in 
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Case(III). For 71,72, 73 € 



have 



a 



(N\G) 



Xz(o, 0,0, 1,1,1) 
Xz(o, 0,0, 1,1,0) 
Xi(o, 0,0, 1,0,1) 
Xz(o, 0,0,0,1,1) 

XZ(0, 0,0, 1,0,0) 
Xz(o, 0,0, 0,1,0) 
Xi(0, 0,0, 0,0,1) 
XZ(0, 0,0, 0,0,0) 



(N\G) if j! ^0,72^0,73^0, (J// x ) 

(i\T\G) i/ 7l ^0,72^0,73 = 0, (7JJ 2 ) 

(N\G) i/ 7l ^0,72-0,737^0, (/// 3 ) 

(iV\G) i/ 7l =0,727^0,737^0, (/// 4 ) 

(iV\G) */ 7l ^0,72 = 0,73 = 0, (/IZ5) 

(7V\G) j/^ =0,727^0,73 = 0, (/// 6 ) 

(iV\G) z/ 7l =0,72 = 0,737^0, (III 7 ) 

(N\G) j/^ =0,72 = 0,73 = 0, (/// 8 ) 



Proof. The case (I). The normalizer Ng{S^) of S*(i) in G is written as the 
semi-direct product Li k 57 where 



A 


2 


2 





Here O2 








A,B G GL(2,R) 



G M(2,R). We define the action of Ng{S(\)) on X(q,7 2 ) as 



( x 'X(a,-y 2 ))( s ) — X(a.-y 2 )( sX ) for x £ Ng(S(j)) and s G 5(1). Then bv lemma l4~8l 
if Xi a ,~, 2 and X/ a /y are m tne same -^Vo(iS , (i))-orbit, the spaces G^ (Si\G) 
and GS° (Si\G) are G-equivalent. Also by Lemma 14. 9[ we only need to 



'T2 



classify the L(/)-orbits of xi a T2 f° r a £ ^\{0} and 72 G M. Then it is easy to 
see that Z( Q ,o.o,o. 72 ,o) G A -d*(A r G(5(i)))(Vi,o.o,o.i)) if 72 7^ and Z( Q ,o.o,o. 72 ,o) G 
Ad*(iV G (5 (I) ))(Vi,o,o,o,o))if72 = 0. 

The case(II). The normalizer of Sm) in G is written as the semi-direct prod- 



uct tx Sun where 



L(n) = 



n {a,b,A) 



= 1 % 






G G I a, 6 G R X ,A e GL(2,K) 



Here O2 = (0, 0) and *02 = ( ^ ) ■ Then there are following L(//)-orbits of 

Xi 01>02 , yin2 for /3i,/3 2 G E (/3 1( 3 2 7^ 0) and 7l , 72 G R, 

Gl = {X{(0,0,i; 1 ,«.»,iui,0,«i 2 ) I +^2^2 7^ 0} 

G 2 = {^(o.o.vi.i^ioiAiDa) I ("1^2) 7^ (0,0), (u>i,w 2 ) 7^ and u^i + t> 2 w 2 = 0}, 
G 3 = {Xi(o : o,t>i^2,t«i,o : «;2) I ( w i> u 2) 7^ (0,0), (wi,W 2 ) = and uiioi + v 2 w 2 = 0}, 
G4 = {Xi(o,o,t>i,'U2,ioi,o ! u'a) I ( w i' u 2) = (0,0), (wi,w 2 ) ^ and uiiui + u 2 iu 2 = 0}. 

The case (III). The normalizer of N in G is written as the semi-direct product 
L(in) x where 



/ ai 



L 



(in) 



«2 



«3 



a 4 J 



Oi, . 



, a 4 G 
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Then the lemma is easily follows. 



□ 



4.2 Differential operators on the generalized Whittaker 
models 

Let U be a closed subgroup of N and \ a character of U. By Theorem 13. 61 the 
space of the generalized Whittaker model is isomorphic to the subspace of 

C™(U\G/K) = {/ G C°°(G) | f(ugk) = X (u)f(g) for (u, g, k) G U x G x X}. 

Lemma 4.11. We retain the above notations. There exists a linear bisection 

3: C™(U\G/K) ^ C X (U\N x A). 

Proof. Because N is a nilpotent group and U is its closed subgroup, there is a 
smooth cross section 9 : U\N — > iV with the smooth splitting of n £ N so that 
n = u(n)s{n) for u{n) £ J7 and s(n) G 6(U\N) (cf. Theorem 1.2.12 in 3 ]). This 
smooth cross section gives us a linear mapping 

3: C™(U\G/K) ^ C°°(U\NxA) 

f — > H(/)(x,a)=/(^)a), 

for x £ U\N and a £ A. Take an element £ C°°(U\N x A). If we define an 
element of / G C™(U\G/K) by 

f$(usak) = x(u)<t>(sa) 

foru£U,s£ 8(U\N) : a£ A and fc G if. Since G = J/xl^TVxAxif, this is well 
defined. We denote this map II. Then it is easy to see that LToS = idc°°(u\G/K) 
and S o n = idc^iuXNxA)- Hence 3 is bijective. □ 

We define the action of [7(g) on C°°(U\N x A) by X ■ 3(f) = 3(R x f) 
for X £ U(q) and / G C™(U\G/K). In this section, we will give the explicit 
expression of the action of U(g) on C°°(U\N x A). 

According to the Iwasawa decomposition g = nffio(Bf, it suffices to see the 
action of n, a and 6 respectively. We can see that En £ a,i = 1, . . . , 4 acts on 
C°°(U\N x A) as $ aj = fli^r, £= 1, ... ,4 if we denote the elements of A by 
a = diag(ai, . . . ,04). By the right if-invariance of C^(U\G/K), the elements 
in { acts trivially. Hence we have the following symmetric relation among the 
generators of the annihilator ideal Ifc(A). 

Lemma 4.12. For F £ C°°(G/K), we have 

((E - Ai)(E -A 3 - k)) tJ F = ((E — Ai)(E - A 2 - k^F, 
where 1 < i,j ' < 4, and fc = 1,2. 

Proof. Take elements (-Ey — Eji), 1 < i < j < 4 as the generators of t. Then 
we have (E l3 - E 0l )F = 0, 1 < i < j < 4 for F £ C°°(G/K), i.e., E^F = 
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EjiF, 1 < i < j < 4. This implies that 
((E-X 1 )(E-X 2 -k)) ij F 

4 

= (J2 E a En - (Ai + A 2 + k)Eij + Ai(A 2 + fc)%)F 

4 

= - (Ai + A 2 + k)Eji + Ai(A 2 + fc)%)F 

2=1 

4 

= {J2(EjiE u - [Eji, Eu\) - (Ax + A 2 + k)E l3 + Ai(A 2 + k)^)F 
i=i 

4 

= {J2{EjiE H - {SuEn - SjiEa)) - (Xi + A 2 + k)E }l + Ai(A 2 + k)S l3 )F 
i=i 

4 

= (J2 EjiEu - {E^ - En) - (Ai + A 2 + fc)^ + Ai(A 2 + fc)<%)F 
i=i 

4 

= (J] EjiEu - (Ai + A 2 + fc)^ + Ai(A 2 + fc)4,)F 

= ((E-Ai)(E-A 2 -fc)) J . j F. 

This is the required equation. □ 

We give more precise expressions of ((E — Ai)(E — A 2 — k))^ mod t/(g)fi for 
k = 1, 2 and 1 < i < j < 4 below. 

Lemma 4.13. Representatives ((E — Ai)(E — A 2 — fc))^ modulo U($)t (k = 1,2 
and 1 < z < j < A) are written as follows, 



Eii + El 



+ E 31 + E 41 



(Ax + A 2 + k - 3)E U 
-(£ 2 + £3 + £ 4 ) + Ai(A 2 + £;), 
E2\{E\\ + £-22 — (Ai + A 2 + k — 3)) + E32E31 + E42E41, 
E 3 i(En + E 33 — (Ai + A 2 + k — 2)) + E32E21 + E 43 E 4 i, 
E 41 (E n + E 44 -(X 1 + X 2 + k- 2)) + E 42 E 21 + E 31 E 43 , 
E22 ~ (^1 + A 2 + — 2)_E 22 +£? 21 + E 32 + E 42 

- (E 33 + E 44 ) + Ai(A 2 + k), 
£'32(^22 + £33 — (Ai + A 2 + k — 2)) + E2\E 3 i + E 43 E 4 2, 
i?4 2 (£ 22 + £44 — (Ai + A 2 + k — 2)) + E21E41 + E 3 2E 43 , 



E 2 33 - (Ai + A 2 + k - 1)E 33 



+ E 3 i+E^ 2 + E\ 3 



E 43 (E 33 +E 44 - (Ai + A 2 + k 



-£ 44 + Ai(A 2 + fc), 
1)) + E 3 iE 4 i + E 32 E 4 2, 



E\ 4 - (Ai + A 2 + k)E 44 + E\ x + E\ 2 + E 2 i3 . 



((i,j) = 


(1,1)) 


{{hi) = 


(1,2)) 


{{hi) = 


(1,3)) 


{{hi) = 


(1,4)) 


{{hi) = 


(2,2)) 


{{hi) = 


(2,3)) 


{{hi) = 


(2,4)) 


{{hi) = 


(3,3)) 


{{hi) = 


(3,4)) 


{{hi) = 


(4,4)) 



Proof. If we note that (Eij — Eji) 1 < i < j < 4 are the generators of 6, this 



lemma can be obtained by the direct computations. 



□ 
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Along the classification obtained in Proposition 14. 101 we express the action 
of n. 

The case (I). We consider the space 

C°° (S (I) \G/K) 

Xi(0,l,e, 0,0,0) v W > ' ' 

Here e = 1 (resp. = 0) corresponds to the case (7i) (resp. (I2)) classified in 
Proposition QUI If we notice that Sp) = {RX 2 + MY 1 + RY 2 + RZ} is not 
only a subalgebra of n but also an ideal of n, then S(i)\n = {RXi + MX3} 
can be seen as a subalgebra of n. Hence S^\N is isomorphic to the subgroup 
{cxp(uXi + vX 3 ) I u, v G R} of TV. This isomorphism gives a smooth cross 
section 9m : Sm\N — > AT. Then we have the linear isomorphism 

S (D : ^(0,^,0,0,0, (S(i)\G/K) - C°°(5 (I) \iV x A) 
by Lemma T4. Ill We introduce a coordinate system on S^\N x A as follows, 

M 2 x (K >0 ) 4 ^ >%)VV x ^ 

(u, v) x (ax, a,2, a 3 , a 4 ) 1 — ► exp(uXi + vX 3 ) x di&g(ai, a 2 , a 3 , a^) 

Proposition 4.14. We regard the space C°°(S^\N x A) as the image of the 
space oQ (S(i)\G/K) by the mapping H(/) for each e = 0,1. TTien n 

acts on C°°(S(j)\N x A) as follows, 

a 2 d , — a 3 
E 21 F = — ^-F, E 31 F = e2ttV^I — ,F 



ai au a± 

E 41 F = 0, E 32 F = 2%V^l— (v - eu)F, 

a 2 

r a 4 1-1 „ „ 04 d 



a 3 dv ' 



E i2 F = 2nV — l—F, E 43 F = — — , 

a 2 

for F G C co (S(i)\N x A). 



Proof. For F G C°°(S {1) \N x A), there exists a / G C£ (0i ., li0i0<0) (S(i)\Gy*Q 
such that F(u,v;a) — 5(i)(/) = /(exp(uXi + wA 3 )a) for u, u G M and a G A 
Hence for £^(1 < j < i < 4) we have 



(E ij F)(u,v;a) = E {I) {R Ei J) 
d_ 
dt" 

d . . , . „ . . .. (4.3) 



— /(exp(uXi + i;X 3 )aexp(t£'y))| 4= o 



= — /(exp(uXi + vX 3 ) exp{tAd(a)E ij )a)\ t= o 

= — —f{exp(uXi + vX 3 ) exp(tEij)a)\ t =o. 

By the direct computation, we have 

exp(uXi + vX 3 ) ■ exp n(z, ■ ■ ■ ,x 3 ) 

= cxp n(z',y[, y' 2 ,0,x 2 ,0) ■ exp((w + Xi)X 1 + (v + x 3 )X 3 ), 
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where 

, 11 111 

z = z + vyi - uy 2 + ^^Vi - T^^ 2 _ UVX2 ~ 2 VXlX2 ~ ~2~ UX2X3 ~ ^ XlX2X3 ' 

Vi =Vi~ ux 2 — , 

/ x 2 x 3 
V 2 = V2+VX 2 + — — ■ 

Hence we have 

f(exp(uX 1 +vX 3 )expn(z,--- ,x 3 )) 

= f{expn(z' ,y[,y' 2 ,Q, x 2 ,0)exp((u + x^Xi + (v + x 3 )X 3 )) 

= XKo,£,i,o,o,o)(expn(z / , y[,y' 2 , 0, x 2 , 0))/(exp((u + x^Xx + (v + x 3 )X 3 )) 

= e 2 ^^ + y^f(exp((u + Xl )X x + (v + x 3 )X 3 )). 

(4.4) 

Combining formulas (|4.3|) and (|4.4[) . we have the proposition. □ 
The case (II). We consider the space 

Xl(0,0,c 1 ,e 2 ,0,e 3 ) v i 11 ^ > ' y 

Here each (£1,62, £3) corresponds to the case (Ilj) (i = 1, . . . ,4) in Proposition 
14.101 as follows, 



(ei,£2,£3) 



(1,1,0) if the case (Hi), 

(0,1,1) if the case (II 2 ), 

(0,1,0) if the case (II 3 ), 

(0,0,1) if the case (II 4 ). 



The subalgebra s (II ) = {«-X 1 + RX 3 + «Y 1 + «Y 2 + M.Z} is a codimension 1 
subalgebra of n. Because any codimension 1 subalgebra of a nilpotent Lie algebra 
becomes an ideal (cf. Lemma 1.1.8 in 3 ), the subalgebra S(n) is an ideal of n. 
By the similar argument as in the case (I), the homogeneous space Sm)\N is 
isomorphic to the subgroup {exp(uX 2 ) \ u £ K} of N. This isomorphism gives 
a smooth section dm) '■ Srn)\N — > N. Then we have a linear bijection 

: C x°? ( o,o, E1 , E2 ,o, E3 )(V)\ G /^ ^ ^(S iU) \N x A). 
We introduce a coordinate system on Sax)\N x A as follows, 

R x (R >0 ) 4 ^ S {U) \N x A 

it x (01,02,03,04) 1 — ► exp(uX 2 ) x diag(oi, a 2 , a 3 , 04) 

Then we can write down the action of n on C°°(Sm)\N x A). 

Proposition 4.15. We regard the space C°° (Snn\N x A) as the image of 
the space C^ (0 c 0e (S(n)\G/K) by the mapping £(//)• Then n acts on 
C°° {S {II) \N x A) 'as follows, 

E 21 F= (2nV=l(—)e 2 )F, E 31 F = 0, 

Oi 

E41F = 0, E 32 F — ——F, 

a 2 au 

E i2 F = (2nV^T( — )ei)F, E 43 F = (2^v^T( — )(e 3 - £l u))F. 
02 03 
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Here F £ C°°(S (II) \N x A) and 



(e\,S2, e 3 ) 



(1,1,0) if the case (II\), 

(0,1,1) if the case (Ih), 

(0,1,0) if the case (Ih), 

(0,0,1) if the case (Ih). 



Proof. The proposition can be obtained in the same way as the case (I) by the 
formula, 



exp(wX 2 ) • expn(^, 
where 



,x 3 ) = exp(n(z , ,y' 1 ,y 2 ,x 1 ,0,x 3 )) -exp(u + x 2 ), 



x\x 2 x 3 



Vi=yi+ xiu 



xix 2 



V 2 = 2/2 - 2:3" - -^x 2 x 3 . 



□ 



The case (III). We consider the space 



C?° (N\G/K) 

Xi(0, 0,0,6! ,e 2 ,e 3 ) v » ' ' 



where 



(eii £2, £3) 



f (1,1,1) 


if the 


case 


(nil), 


(1,1,0) 


if the 


case 


(III 2 ), 


(1,0,1) 


if the 


case 


(Ills), 


(0,1,1) 


if the 


case 


(III4), 


(1,0,0) 


if the 


case 


(III 5 ), 


(0,1,0) 


if the 


case 


(III 6 ), 


(0,0,1) 


if the 


case 


(III7), 


,(0,0,0) 


if the 


case 


(His)- 



By the Iwasawa decomposition, we have the linear bijection 



-(in) 



C 



X!(0,0,0,ei,e 2 ,£3) 



(N\G/K)3 f» f\ A £C°°(A). 



by the restriction to A. Then we have the following proposition. 



Proposition 4.16. Let us consider the space C°° (A) as image of the space 
C™ oooe e e (N\G/K) by the mapping £(///). Then n acts on C°°(A) as 
follows, 



E21F = (2tt 
E 41 F = 0, 
E i2 F = 0, 



ai 



E 31 F = 0, 



03- 



E 32 F = (2irV^l(-)e 2 )F, 

a-2 
»4 , 



E i3 F = {2W-l(—)e 3 )F. 

a-3 
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Here F € C°°{A) and 



(ei, £2, £3) 



= < 



(1,1,1) 
(1,1,0) 
(1,0,1) 

(0,1,1) 

(1,0,0) 

(0,1,0) 

(0,0,1) 
1(0,0,0) 



Proof. It is obvious from the formula, 



if the case (Hh), 
if the case (III2), 
if the case (Hh), 
if the case (Hh), 
if the case (Ilh ), 
if the case (Ills), 
if the case (III7 ), 
if the case (111%). 



(E ij F)(a) = ^F(aexp(tE tj ))\ t=0 = — F(exp(iAd(a)£^)a)| i=0 . 



for 1 < i ^ j < 4. 



□ 



4.3 Generalized Whittaker models of GL(4,R) 

After these preparations given in the previous sections, we can investigate ex- 
plicit structures of the generalized Whittaker models. More precisely to say, we 
will give the dimensions of these spaces and their basis in terms of the hyper- 
geometric functions one and two variables. 



4.3.1 The embeddings into the spaces (I) 

We will give the explicit structures of the embeddings of the Harish-Chandra 
modules Xk : \ (k — 1,2) into the spaces (±1) and (I2) which are classified in 
Proposition ^. 101 These spaces are isomorphic to C^ (o ± E goo) (S^\G/K ; hW) 
by TheoremEU We will consider the image of C~ (o ' o ' {S (l) \G/K ; I k {\)) 
by the mapping Hp) defined in Section FOl Here e = 1 (resp. = 0) for (Ii) 
(resp. (12))- Hence our purpose is to investigate these spaces 

C x l(0 ^oJS m \N x A; / fc (A)) = ^(C^^JS^G/K; I k (X))). 

Proposition 4.17. For e = 0, 1, we consider the spaces x E 0) {S(i)\N x 
A; /fe(A)). Then these are the solution spaces of the following systems of the 
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differential equations on C°°(S(j)\N x A). 

[0* -(Ai + A 2 + k - 3)0 O1 + Ai(A 2 + fe) + (— ) 2 ^ 



_ ( 4 - 5 ) 
+ e(^) 2 (2 7 r % /^T) 2 - {§ a2 + ti a3 + § ai ) + Ai(A 2 + fc)]0 = 0, 
01 



[«-(i?ai + <?a 2 - (Ai + A 2 + k - 3)) + e(^) 2 (27t\/=1)> - eu)]<j> = 0, (4.6) 
ou o 2 

[e^ + tfa 3 - (Ai + A 2 + k - 2)) + (v- eu)^ = 0, (4.7) 



(4.9) 



K, ~(\ 1+ \2 + k- 2)tf Q2 + (^) 2 |^ + (^) 2 (2tt^) 2 (« - euf 

a\ ou^ a 2 

+ (— ) a (27iV=l) 2 ~ + $a 4 ) + Ai(A 2 + k)]<f> = 0, 

[(« - eu)(^ 2 + tf 03 - (A! + A 2 + k - 2)) + e(^) 2 |- + (-) 2 |-]</> = 0, 

oi ou a3 

(4.10) 

[(0a, + ^a 4 - (Ai + A 2 + fc - 2)) + (« - eu)-|-]0 = 0, (4.11) 

av 

[<,-(Ai + A 2 + fc - l)0 a , + e(^) 2 (2^V^T) 2 

Ol 



f (^) 2 ( 27 rV=T)> - euf + (^) 2 |^ - D ai + A X (A 2 + = 0, 
a 2 a 3 a?r 



(4.12) 



a -(i?«s + ^ 4 - (Ai + A 2 + fc - 1)) + (^) 2 (2tt\/=T)> - eu)]<j> = 0, (4.13) 
ov a 2 

q 2 /-v | \ . ; \ q , / a 4\2/n / — T\2 , / a 4-, 2 ® 2 



[< - (Ai + A 2 + k)V ai + (^y(2W-iy + (-y^ + Ai(A 2 + *)]</> = 0, 

a 2 a 3 c>ir 

(4.14) 

[(0 O1 + #a 2 + ^ 3 + ^a 4 - fcAi - (4 - fc)A 2 )]0 = 0. (4.15) 
Here <f> € C°°(S (I) \N X A). 

Proof. Recall that /fc(A) is written as (|4.1|) . Then these differential equations 
immediately follows from Lemma I4.12[ Proposition 14.131 and Proposition 14.141 

□ 

We solve these systems of the differential equations, 
(i) The case e = 1. 

We investigate the space C^ (o 1 1 0) (S(i)\N x A; i&(A)), i.c, the case e = 1. 

We consider the case k = 1, i.e., the embedding of the Harish-Chandra 
module A^a- 

Proposition 4.18. For £/ie case e = 1 and k = 1, the solution space of the 
system of the differential equations defined in Proposition ^. ll\ is {0}. 

Proof. The equations (|4.7[) and (|4.11[) give us the equation, 

d d 

[#ai + $a 2 + $a 3 + #a 4 - 2Ai - 2A 2 + 2 + 2(v - «)(— + — )]0 = 0. 
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By the equation (|4.8p , the term + can be eliminated. Hence the only 
remaining term is 

[0 ai + $a 2 + $ a3 + ^a 4 - 2Ai - 2A 2 + 2]<j> = 0. 
However if we compare this with the equation (|4.15p . 

[# ai + $a 2 + $a 3 + $a 4 - A X - 3A 2 ]0 = 0, 

we can conclude the solution space of these equations must be {0} because we 
assume Ai — A 2 ^ Z. □ 

Next, we consider the case k — 2, i.e., the embedding of the Harish-Chandra 
module X% \. We introduce a new coordinate system below, 



X2 = (?rV^ 



^3 



X\ = aio 2 a 3 a4, 

'^i) 2 _|_ (^) 2 
v a 2 ai 
\ -2 



T) 2 ( 


a 2 


-u) 2 + 


aia 3 


{v-uf 


a 2 a 3 


a 2 a4 






X4 = 


aia 3 
a 2 a 4 ' 




x 5 = 


0104 
a 2 a 3 ' 




x 6 ■■ 


= u. 



(4.16) 



Proposition 4.19. Let e = 1 and k = 2. We consider the system of the differ- 
ential equations in Proposition ^- 1 7\ By adding and substituting each differential 
equations and multiplying some rational functions, the system of the differential 
equations under the new coordinate system written as followings , 

(^i-^^^ ' (4 ' 17) 
[*2 ~ (^2 - ^)(20», - ^ 2 )]0 = 0, (4.18) 



[x 3 (^ 2 -2^ 3 )(^ 2 - 2^-1) 

- (^3 ~ J(A 2 - Ai) - l)^ + ~(A 2 - Xi)M = 

9X4 



(4.19) 



d 
dx 5 

d 
dx 6 



0, (4.20) 
0, (4.21) 
0. (4.22) 



Proof. First, we put 
12. 

a4 = a 3 a4 , u' — u, v' — (v — u). 



ax = aia 2 , a 2 = aia 2 , a 3 = a 3 a 4 , 
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Then the differential equation (|4.8p becomes 

A^ = 0. (4.23) 

Furthermore we exchange the variables a 2 , a±, v' to 

w = o^a^v' 2 + a2«4 1 + a 2 " 1 a4, /3 2 = 1220:4, /?4 = 0.201^ ■ 

Then equations (|4.10p and (|4.1ip become 

%0 = 0, (4.24) 
0^ = 0, (4.25) 

respectively. Setting 

/3i = aia 3 , /3 3 = aaa^ , 

the equation (|4.15p becomes 

(20ft - (Ai + A 2 ))«£ = 0. (4.26) 
Also we can see that the equation (|4.6j) is written as 



[2w^-(2(% + %) - (Ai + A 2 - 1)) - (2nV^l) 2 Ps 1 M^ = 0- (4.27) 

If we eliminate 0ft from (|4.27|) by using the equation (|4. 26(1 . it can be written 
as 

d 

[2w— (20ft + 1) - {2tt^I) 2 Pz l w](t> = 0. (4.28) 



dw 

We note that the equation (|4. 13|) can be reduced to the same equation. Taking 
into account the equations (|4.26p and (|4.28p . the equation (|4.5p can be reduced 
to the equation 

[(0ft + K - \(Xx - A 2 - 4))(% + d w + i(Ai - A 2 )) - 4^]0 = 0. (4.29) 

We can also see that the equations (14.91) . (|4.12p and (|4. 14[) can be written as 
the same equation (I4.29p . Finally, we put 

71 = (tt v ^1) 2 (3^ 1 w, 72 = w" 2 . 

Then the equation (14.28P is equivalent to 

[(0 7i -20 72 )(i-0 71 )- 7l ]0 = O. (4.30) 

Also the equation (|4.29|) is written as 

[(0 72 -\(X»- Ai) - 1)(0 72 + |(A 2 - Ai)) - 72 (0 71 - 2tf 72 )(0 7l - 20 72 - 1)]0 - 0. 

(4.31) 

If we put 

xi = /3i, x 2 = 71, a; 3 = 72, 

x 4 =/3 2 , x 5 — Pi, x 6 = u', 

then the theorem follows from the equations ([423]) . (|4~24]) . ff4~25]) . (|4T26]) . (|430]) 
and 1(431]) . 

□ 
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Corollary 4.20. The change of variables J4.J6'[ j gives a diffeomorphism from 
{(ax, . . . , CI4, u, v) el 6 | ai G R>o (i = 1, • • • , 4)} = Sm\N x A to the domain 
D x = {(x-l,...,x 6 ) I Xi G R>o (i = 1, 3, 4, 5), x 2 € R<o, ^6 € R}. 

Proof. We should show that this gives bijection and the Jacobi determinant is 
not zero, i.e., 



d(xi, ...,x 6 ) 



d(ai,...,ai,u, v) 



(P) = 



few ■•■ TO TO 



3xi , 



9x1 



••• to to to 



for any j? G {(ai, ■ ■ • , 04, u, v) G R 6 | Oj € R>o (i = 1, . . . , 4)}. Here \X\ means 
the determinant of X G M(6, R). As we see in the proof of the previous propo- 
sition, this change of variables is the composition of the following change of 
variables. 
Stepl. 



Oil = aiQ2, 



ai = a 3 a 4 , 



o?2 = a\a 2 
v! = u, 



a 3 = a 3 a 4 , 
v = (v — u). 



Here we can see this gives a bijection from {(cti, . . . , <X4, u, v) G R 6 | aj G 
R >0 {i = 1,...,4)} to {(ai, 
Step2. 



,04, «',«') gR 6 I a, gR >0 (i = l,...,4)}. 



/?4 = a 2 a 4 " 1 , 



/3 3 = aia 3 \ 



/2 —1 —1 

w = a-ia^v + 0:20:4 + a 2 on, 



Here we can see this gives a bijection from {(ai, . . . ,on,u' ,v') G R 6 | Oj € 
R >0 (i = l,...,4)} to ft,»,«')eK 6 I ft,weR> (t = l > ...,4)}. 

Step3. 



x\ = Pi, 
Xi = p 2 , 



X2 = {TT^lfP^W, 

X 5 = Pi, 



X3 = w 
Xq = U . 



Here we can see this gives a bijection from {(Pi, ■ ■ ■ ,Pi,w,u') G R 6 | Pi,w G 
R >0 (t = 1,...,4)} to {{x x ,...,x' 6 ) G R 6 I Xi G R> (« = 1,3,4,5), x 2 G 
R <0 ,a; 6 GR}. 

Also it is not hard to see that 



d(xi, ...,x 6 ) 



(P) 



d(ai, . . .,ai,u,v) 

d(xi, ...,x 6 ) d(Pi, . . .,Pi,w,u') d(ai, . 



for any p G {(ai, . . . , ai, u, v) G R 6 | G R>o (i = 1, 
proposition. 



, ai, v! , v 1 ) 



d(Pi, . . . , Pi, w, u') d(a\, . . . , ai, u' , v') d(a x , . . . , 04, it, v) 



(P) ^ 0, 



,4)}. Thus we have the 
□ 



Pick up the differential equations (|4.18p and (|4. 19jl . We take f(x%,x^) as a 
solution of them. We take a function F(x%, X3) such that 



/ = ^2 F. 
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Then this F(x 2 ,xs) satisfies 

[x 2 - $ X2 (20 a , - t? aa + i(Ai - A 2 - l))]F(x 2 , a; 3 ) = 0, (4.32) 

[a!3(20xa " + £(Ai - A 2 - l))(2i? S3 - $ X2 + i(A x - A 2 - 1) + 1) 

2 2 (4.33) 

-0* 3 (tfx 3 + g ( A l - A 2) - 1)MX2,X 3 ) = 0. 

These are the differential equations for Horn's hypergeometric function Hio(^(Ai- 
A2 — 1), s(Ai — A 2 ); X2, X3) (cf. [H]). We denote it by Sj 10 (a, d; x, y) the solution 
space of the system of the partial differential equations for Horn's hypergeomet- 
ric function Hio(a, d; x, y), i.e., 

[x{2d x -d y + a){2d x -§ y + a + l)-$ w {'& x + d- l)]f(x, y) = 0, 
[y-& y (2d x -# y + a)]f(x,y) = 0. 

We can see more detailed properties of Sj 10 (a, 0?; x, y) in Appendix. 

Definition 4.21. Let U C K" &e a domain. A function f(x) on U is called 
rapidly decreasing on U if it satisfies, 

sup \x a f(x)\ < 00 

x£U 

for any a — (ai, . . . , a n ) £ N n , where x a = x" 1 ■ ■ ■ x" n . 

Also a function f(x) on U is called slowly increasing on U if there exists 
N e N such that 

sup(l + \x\)- N \f(x)\ < 00 
xeu 



where \x\ = \J x\ + ■ ■ ■ + x\ for x E U. 

Theorem 4.22. The space C^ (o ll00 (S(i)\N x A; 4(A)) consists of the el- 
ements, 

x x 2 x^x^ 1 2 f(x 2 ,x 3 ), 
for f{x,y) £ ^io(|(Ai - A 2 - 1), |(Ai - A 2 );a;,y). Here 

xi = aia 2 aza^, 

x 2 = (7rV-l) [{ — ) (v-u) +(— ) +(— ) 
\ a 2 a 2 01 

x 3 = (u -it) + 

\a2a4 0104 a2«3 

aia 3 

x 4 = , 

a 2 a4 

X5 = , 

a 2 a 3 

Xq = U. 

Thus we have 

dimc^ (0|lilOi0|O) (5 w \iVxA;7 fe (A))=4. 

In C^ (0 j 1 , (S(jj\N x A; Ik(X)), there is a rapidly decreasing function on 
{{X2, ■ ■ ■ ,Xq) I x 2 € K<o, £3, x±, X5 S ]R>o,2:g € M} and if is unique up to 
constant multiple. 
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Proof. This follows immediately from Proposition ^. 19l and the argument above. The 
second assertion follows from Appendix B and Thorem lB.ll □ 

(ii) the case e = 0. 

We investigate the space C^ (o looog) (S^\N x A; 7&(A)), i.e, the case e = 0. 
We introduce a new cooridinate system as below, 



X\ = ai, 

x 3 = a\ + a\v 2 , 

x 5 = a 3 a;7 \ 



X2 = a 2 , 

I — 2 1 -2 2\-l 

x 4 = (a 3 + a 4 v ) , 
= u. 



Lemma 4.23. The change of variables given above is the diffeomorphism from 
{(ai, . . . , a 4 , u, v) € E 6 | a 4 e R >0 (i = 1, . . . , 4)} = S^^\N x A to the domain 
D 2 = {(xi, . . . , x 6 ) I Xi G M> {i = 1, ■ • • , 5), x 6 e M}. 

Proof. We can see that this is a bijection. Also it is not hard to see that 

<9(cci, . . . ,x e ) 



d(ai, . . . , 0,4,11, v) 



(P) ^ 0, 



for p e {(01, ...,a4,«,i))£l 6 a; G M>o (i = 1, . . . , 4)} by direct computation. 

□ 

Then the systems of the differential equations in Proposition ^. 171 are reduced 
to the followings 

Proposition 4.24. For k — 1,2, we consider the systems of the differential 
equations in Proposition ^. 17\ Then by adding and substituting each differential 
equations and multiplying some rational functions, these systems are written on 
{(ai, . . . , 04, u, v) € R 6 I aj S E>o (i = 1, . . . ,4)} as follows. 



(0»x-(Ai-(4-fc)))(0 xl -A 2 )^ = O, 



[tf 2 2 - (Aj + A 2 - 3 + fc)^ 2 + B1 



(4.34) 
(4.35) 

(4.36) 

(4.37) 
(4.38) 
(4.39) 



[< - (A x + A 2 - 4 + fc)0 Ol + [-11 — + A 2 (A a - (4 - fc))]tf = 0. 

By the equation (|4.8[) . we can eliminate the term ^ from the above equation, 
then it can be written as 



+ (2ttV^1) 2 Xz 2 x 3 + A 2 (Ai - (4 - fc))]0 = 0, 

[4« + O - 2 ( A i + A 2-! + fc )^3 + 2 ^4 

+ (27rV^T) 2 a; 2 " 2 a;3 + 2Ai(A 2 + k)](f) = 0, 
(2tf X4 -Ai)(2tf X4 -(A 2 + fc))<£ = 0, 
^.0 = 0, 
0^0 = 0. 

Proof. By the equations (|4.5p and Q4.15JI . we have the new equation 

'a 2 \ 2 9 2 



(tf.i-(Ai-(4-fc)))(l? 01 -A 2 ) 



0. 



(4.40) 
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Next, from the equations (|4.9[) and (|4.15[) . we have a new equation 

[^-(X 1 +X 2 -3+k)^ a2 +^ ai + (2nV^T) 2 (al+alv 2 )a^ 2 + \ 2 {X 1 -(4-k))}^ = 0, 

(4.41) 

as well. If we put = 0304 and 04 = 0304 , we have a new equation 

from the equation (|4.10j) and ()4.11j) . Moreover if we put w = a 4 1 + a^v 2 , 
fh = a 3 and /?4 =014, the above equation is reduced to 

0/j 4 - 0. (4.42) 

And the equation (|4.13p becomes 

1 



[MMfk - (Ai + A 2 - 1 + k)) + ~(2nV-iy/3awa^]<j> = 0. (4.43) 

Also if we consider the sum of the equations (|4.12[) and (|4.14[) . we have a new 
equation 

[2d} 3 - 2(Ai + A 2 + k)tif3 3 + 2$ 2 W + 2§ w + (W=I) 2 tfl&oJ 2 + 2Ai(A 2 + k)]tf> = 0. 

(4.44) 

By the equation (|4.43p . we can eliminate the term fl^wa^ 2 from (14.44p . Then 
we have 

(0ft - $ w ~ Ai)(0ft -0 W - (A 2 + fc))0 - 0. (4.45) 

Hence if we put 

x\ = at, x 2 — a 2 , x 3 = /3 3 u;, (4.46) 

.t 4 = /3 3 w~\ z 5 = /3 2 , x 6 = u, (4.47) 

then we have the proposition from the equations (14.40p . (14.411) . (I4.42p . (I4.44p . 
(jOST) and gU). □ 

Let 97t2$(z; ; a;) be the solution space of the differential equation 
, d 2 Id . v 2 , . , 

b^ + -^-( 1 + -)]/( a; ) = ' 

dec 2 a; ax x z 

1. e., the solution space of the modified Bessel equation. We have dime 3712$ (f ; x) - 

2. In 07lQS(i/ ; x), there is a series solution 



0° ( x\v+m 



— ' m!r(i^ + m + 1) 



Also there is a solution as a slowly increasing function 

= 7T -J,(X) 

2 sin i/7r 

and any slowly increasing function in 9712$ (1/ ; x) are constant multiples of K v (x). 
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Theorem 4.25. For the cases k = 1,2, C°° (S m \N x A: 4(A)) are 

' ' X!(0, 1,0, 0,0,0) v ( J / \ ' 

written as follows under the coordinate system, 

x% = a%, x 2 = a 2 , 

2 2 2 / 2 2 2\ 1 

x 3 = a 4 + a 3 v , X4 — (a 3 + a 4 v ) , 

X5 = a3«4 , x e = u. 

(i) For k = 1, C% [OMOim (S (I) \N x A; 4(A)) connate o/ 

Ai+A 2 -2 \i+X 2 A 2 + l 

a;^ 2 a; 2 2 ir 3 4 x 4 2 /(27rx 2 ' 1 % /a^') 
/or f(x) G 9Jt<B( A i-^-2 . ^ Thug we have 

dim c C°° (Sm\N X A; 4(A)) = 2, 

"~ X!(0, 1,0, 0,0,0) v (v > * ' 

and i/iere is a slowly increasing function on {{x±, . . . ,xq) \ xi G R>o (i = 
1, . . . , 5), xq G K} and it is unique up to constant multiple. 

(ii) For k = 2, C™ 01 ^ o o o) (S (I) \N x A; 4(A)) coruufo o/ 

Xl + X 2 -1 Xl+A 2 + 1 X]_ 

C^ 2 x 2 2 x 3 4 £ 4 2 /(2ttx- 1 v /^') 

Aj+A 2 -1 A!+A 2 + l A 2 +2 

+ C"x Ai_2 :e 2 2 x 3 4 x 4 2 3(2 7 ra; 2 " 1 v ^), 

w/iere C, C" G C, /(i) G SmQ3( Al ~ A2 ~ x ; z) anrf G <mff( Al ~ A2 ~ 3 ; x). 17ms 
we /iaue 

dim « ^0,1,0,0,0,0) {S m \NxA;I a (X))= 4, 
and £/iere is a 2- dimensional subspace which consists of slowly increasing func- 
tions on {(xx, . . . , xq) \ Xi G R>o (i = 1, . . . , 5), xq G M}. 

Proof. The solution space of the equation (|4.34|) consists of the elements 

Ci0i(a;2,a;3,a;4)a; Al_(4_fe) + C 2 2 (a; 2 , #3, x A )x^ 2 , 

where Ci are constants and are arbitrary functions for i = 1,2. We determine 
functions fa by the equation (|4.35p . Then these functions satisfy following 
equations, 

[K, - (Ai + A 2 - 3 + k)d X2 + (2nV^l) 2 X2 2 x 3 + (Ai — 4 + fc)(A 2 + l)]fa = 0, 
Wt 2 - (Ai + A 2 - 3 + k)d X2 + {2nV^l) 2 X2 2 x 3 + (Ai - 3 + fc)A 2 ]0 2 = 0. 

A l+A 2 -3+fc 

We define the functions fa so that fa — x 2 2 0^ for i — 1,2, then ^ satisfy 
following equations, 

[<, - ((2^ 2 " 1 V^) 2 + ( Al ' A2 2 ' 5 + fc )2M = °" 

- ((2^ 2 - 1 v^) 2 + ( Al " A2 2 " 3 + fc ) 2 M = 0. 

For some fixed x 3l if we put z = (Owx^ *Jxs), these equations are nothing but 
modified Bessel equations 

r d 2 Id , vf . . . 
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where = Al ~ A |~ 5+fc and v 2 = Al ~ A |~ 3+fc . Hence the intersection of the 
solution space of the equations (|4.34[) and (|4.35|) consist of functions written as 
follows, 



OiCi(x 3 ,Xi)fi(x2,x 3 )x^ 1 (4 k) x. 



2 

2 

A l+ A 2~3+k 



+ C 2 (2{x3,X4,)f2(x2,X3)Xi 2 X 



2 



where Ct are constants, Q are arbitrary functions and fi G 9JtQ3(z^ ; 2ttx 2 1 \fx~z) 
fori = 1,2. 

By the same argument, we can also see that the intersection of the solution 
spaces of the equations (|4.36p and (|4.37j) are written as follows, 

Aj+A 2 — 1+fe Xi 

Di^i(xi,X2)gi{x2,x 3 )xl 4 x 4 2 

\ 1 +\ 2 -l + k A 2 + fc 

+ D 2 £,2{xi,X2)g 2 (x2,x 3 )x 3 4 x 4 2 , 

where Di are constants, £j are arbitrary functions and ^ G 9!7lSS(/i.j ; 27rx 2 ~ 1 y/x3) 
for i = 1,2. Here we put /ii = Al ~ A2 2 +1 ~ fc and /i 2 = Al ~ A |~ x ~ fc . 

Note that $H*B(^ ; x) D £DtQ3(/i ; x) = {0} if z/ 7^ /1. Then intersections of the 
solution spaces of the equations (|4.34p , (|4.35jl , (|4.36|) and ()4.37[> are written as 
follows. For k = 1, 

A!+A 2 -2 A t +A 2 A 2 + l 

x l 2 x 2 2 x 3 4 x 4 2 f(2nx2 y/xg) 
where /(x) € 9Jl i B( Al ~ A2 ~ 2 ; x). And for fc = 2, 

A! + A 2 -l A!+A 2 + l Al. 

Cx A2 x 2 2 x 3 4 x 4 2 /(2 7 rx 2 - 1 v / i3) 

, \ _ 2 A l+ A 2-1 A l+ A 2 + 1 \ 1 2 _1 

+ Cx 1 1 x 2 2 x 3 4 x 4 g(2irx 2 y/xs), 
where C*, C € C, /(x) € 97l ( B( Al ~ A2 ~ 1 ; x) and 5 (x) £ 9JtQ3( Al ~ A2 ~ 3 ; x). □ 

4.3.2 The embeddings into the space (II) 

We consider the embeddings of Xk,\ (fc = 1,2) into the spaces (II j) (i = 
1, . . . ,4) which are classified in Proposition 14. 101 These spaces are isomorphic 
to C x((o,a,e 1 ,e 2 ,o,e 3 ))( S m\ G / K ' We consider the image of this space by 

the map sm-j defined in Section |4~21 Here 



(ei,£2, £3) 



(1,1,0) if the case (IT), 

(0,1,1) if the case (II 2 ), 

(0,1,0) if the case (II 3 ), 

(0,0,1) if the case (II 4 ). 



We denote these space by 

C Zo, OM .J S (u)\N x A; 4(A)) = E m (C~ 0i0it ^S^G/K; 4(A)). 



39 



Proposition 4.26. The function space C^ (o o £ s o e } (S(uj\N x A; /fc(A)) /or 
fc = 1, 2 are eguaZ to i/ie solution spaces of the following systems of the differen- 
tial equations in C°°(S(j]j\N x A). 

[<-(A!+A 2 -3 + fc>V + (2W^T) 2 (-) 2 £2 , x 

oi (4.48) 

+ Ai(A 2 + k) - (l?a 2 + 1>03 + = 0, 

+ ^a 2 - (Ai + A 2 - 3 + fc))0 = 0, (4.49) 
^0 = 0, (4.50) 
eie 2 ^ = 0, (4.51) 

(4.52) 



[< - ( Al + A 2 - 2 + fc)tf a2 + (2^) 2 (^) 2 £ 2 + (^) 2 |^ 

+ (2vrV Z T) 2 (^) 2 e 1 + A X (A 2 + fc) - (tf 03 + ^)]<j> = 0, 
a 2 

r a 3 9 / a , a /x , x o , i.w , /n_ I — 7\2 a 4 



[— p-(0a 2 + ^a 3 - (Ai + A 2 - 2 + k)) + ^V-iy—e^es - e lU )]0 = 0, 
a 2 au 0203 

(4.53) 

[ei^aa + ^a 4 - (Ai + A 2 - 1 + k)) + (0 3 - t?iu)^-]0 - 0, (4.54) 



[<-(Ai + A 2 -l + ^ a3 + (^) 2 |^ 



(4.55) 



+ (2W=Tr(^r( £3 - W - *«u + Ax(A 2 + k)]4> = 0, 

03 



[(e 3 - £i«)(i 3 + ??a 4 - (Ax + A 2 - 1 + fc)) + = 0, (4.56) 



[< - (Ai+A 2 + k)d a , + (2nV^l) 2 (-) 2 £i 

+ (27r^) 2 (^) 2 (e 3 - e lU f + Ai(A 2 + fc)]0 = 0, 

03 



(4.57) 



+ ^a 2 + 0«, + - k\i - (4 - fc)A 2 ]0 = 0. (4.58) 
Here <p € C°°(S( U )\N x A). 

Proof. As well as Proposition ^. 171 these are obtained by the direct computation 
from Lemma 14.121 Proposition 14.131 and Proposition 14.151 □ 

(i) The case (ei,£ 2 ,£3) = (1,1,0). 

Theorem 4.27. When (£i,£ 2 ,£ 3 ) = (1,1,0), we have 

C x l ^,o, o ^ (u) \NxA-,I k (X)) = {0}, 

for k = 1,2. 

Proof. It is immediate from the equation (|4.51[) . □ 

(ii) The case (£i,£ 2 ,£3) = (0,1,1). 

Theorem 4.28. When (£i,£ 2 ,£ 3 ) = (0,1,1), the space C~ o o o _ w) (S (U) \N x 
A; ife(A)) (fc = 1,2) are written as follows. 
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(i) If k — 1, we have 

c x l(0 , , , 1M (S (I r ) \NxA-,i 1 (x)) = {o}. 

(ii) Ifk = 2, the space C™ (0 ^ 0101) (S (II) \N x A; J 2 (A)) consists of 



x 1 2 x|x 3 2 x\ f(2nx2)g(2nx 3 ) 

for f(x),g(x) e MB( Xl ~2 2 ~ 2 \ x )- Here we put 

X\ = aia2, xi — a^ 1 a2, Xs — 0304, X4 — a 3 ~ 1 a4. 

Thus we have dime o i o (S (jj)\N x A\ /2(A)) = 4. There exists a 

slowly increasing function on the domain {(x\, . . . ,0:4,1*) | Xi G R>o,u € K} 
^xuo 00101) {S(n)\N x A; /2(A)) and if is unique up to constant. 

Proof. We show the case k = 1 first. By the equations (|4.49[) and (|4. 56[) . we 
have the new one, 

[0 OI + #a 2 + ^a 3 + ^a 4 - 2Al - 2A 2 + 2]0 = 0. 

Comparing this equation with the equation (14. 58[) . we can conclude that the 
space C^ (o o o i o (S(ii)\N x A; Ii(X)) is equal to {0}. Next, we consider the 

case k = 2. By the equation (|4. 50(1 . we can eliminate the terms from the 
other differential equations. Then the equations in Proposition ^. 26l are reduced 
to the followings, 

[0 O1 + - (Ai + A 2 - l)]cj> = 0, 
[<-(A 1 +A 2 -l)i? ai + A 1 (A 2 +2) 

+ {2TiV^i)a^ 2 al - (i? Q2 + tt , + t9 ai )]0 = 0, 
l€ 2 - (Ai + A 2 R 2 + (2^V^T) 2 + Ai(A 2 + 2) - (i? 03 + tf a4 )]<£ = 0, 

[K 3 + $ ai ~ (Ai + A 2 + l)]<j> = 0, 
[tf 2 3 _ ( Al + A 2 + l)tf a3 + (2^x/^T) 2 a3: 2 a 2 _ $ ai + Ai(A 2 + 2)]o> - 0, 
K - ( A i + A 2 + 2)tf Q4 + (27rV^T) 2 a^ 2 a 2 + Ai(A 2 + 2)]0 = 0. 



We put 



xi = aia 2 , x 2 = a x a 2 , 

X3 = 0304, X4 = a 3 " 1 a4, 



Then we can rewrite above differential equations as follows, 



[2i? Sl - (A x + A 2 - l)](t> = 0, 
[2tf X3 - (Ai + A 2 + l))4> = 0, 



(2tt> 
(2tt> 



4x 2 ) 2 - ( 
4X4) 2 - ( 



Ai — A2 
2~ 

Ai — A2 



4 JV 
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We take <j>' as <\> = x\ <j>' ' . Then we can see that <fr' satisfies following equations, 

[20 X1 - (Ai + A 2 - 1)]^' - 0, 
[2^3 - (Ai + A 2 + 1)]^' = 0, 

K - ((2vV=lx 2 f + ( Al " X 2 2 ~ 2 ) 2 )W = 0, 
- ((2nV-lx 4 f + ( Al ~g a ~ 2 ) 2 )]0 = 0. 
Then we can conclude that 

Ai+M-l 1 A! + A 2 + l i 

(f)(x 1 ,x 2 ,x 3 ,x 4 ) = x 1 2 x|x 3 2 xf f(2TTx 2 )g(2Trx 3 ), 

where f,g £ 9JIQ3( Al ~ Aa ~ 2 ; x). Hence we have the proposition. □ 

(iii) 27ie case (ei,£ 2 ,£3) = (0,1,0). 

Theorem 4.29. Mien (ei,e 2 ,e 3 ) = (0,1,0), fee space tf£ (0i0i0>li0i0) (S<7z;VV x 
A; Ifc(A)) (fc = 1,2) are written as follows. 

(i) Ifk=l, the space C^ i{oooioo) (S (II) \N x A; h(X)) consists of 

ai+a 2 -2 1 
X! 2 x 2 2 (a;3a;4) A2+1 /(27ra; 2 ) 

/or f{x) G 97133 ( Al ~ A2 ~ 3 ;:c). tfere we put 

xi = aia 2 ,a; 2 = a^ 1 a 2 ,x 3 = 03,2:4 = a 4 . 

Thus we have dime C^ (0 x 0) {S(n)\N x A; ii(A)) = 2. There exists a 
slowly increasing function on the domain {{x\, . . . , £4, it) | xi € K>o, w € tw 
C^ (o o Q i Q (S(jjj\N x A; 7 2 (A)) and it is unique up to constant. 

(ii) If k = 2, ffce space C£° ooo i o {S (II) \N x A; J 2 (A)) consists 0/ 

(Ci^ 2 ^ 1 +C 2 ^ 1 ~ 1 4 2+2 ) x x^ 2 ' 1 x\f{2irx 2 ) 

for f(x) G <H j B( Al ~^ 2 ~ 2 ;a:) and d, C 2 G C. Here x l (i = 1 . . . , 4) are same as 
ft- 

T/ims we Ziawe dime C^ (0 1 0) {S(n)\N x A; / 2 (A)) = 4. There exists a 2- 
dimentional sup space o/C^ (o o g i o o (S(jjj\N x ^4; I 2 (A)) which consits of slowly 
increasing functions on {(xi, . . . , £4, it) | £&j € K>o, u € R}. 

Proof. By the equation (|4.50[) . we can eliminate the term -J^ from the other 
equations. Then the equations in Proposition ^. 26l are reduced to the followings, 

[< - (Ai + A 2 - 4 + fe)i? ai + {^^Ifa^al + X 2 (X 1 - (4 - fc))]0 = 0, 
[# 01 + i?a 2 - (Ai + A 2 - 3 + k)]<f> = 0, 
[< - (Ai + A 2 - 3 + £0tf a2 + (2^V^T) 2 or 2 a 2 + O1 + A 2 (Ai - (4 - k))]<j> = 0, 
[< - (Ai + A 2 - 1 + fc)i? Q3 - d ai + Ai(A 2 + fc)]</> = 0, 
[(tf O4 -Ai)(tf Q4 -(A 2 +fc))]0 = O. 



42 



We put 



xi = aia 2 , x 2 = a 1 1 a 2 . 



We take <j>' as <f> = x 2 4>' . Then we have 



- ((2™ 2 ) 2 + ( Al A % (4 fc) ) 2 M = 0, (4.60) 

[< - (Ai + A 2 - 1 + fc)i? a3 - ^a 4 + Ai (A 2 + k)]4>' = 0, (4.61) 
( l ? a4 -A 1 )(i9 a4 -(A 2 + fc))0' = O. (4.62) 

The solution of (|4~59]) and (1460]) is 

A l + A 2 -3+fc 

0'(a;i, ar 2 ,03) 04) = c^ja^X! 2 f(2-Kx 2 ), 

for an arbitrary function 0(03,04) and f(x) £ Wl%$( A i~ A 2-(4-fc) . ^ g0ive 
the equations (|4.6ip and (|4.62[) to determine the function 0(03,04). Then we 
can see that 



c(a 3 ,a 4 ) 



[(a 3 a 4 ) A2+1 fork=l, 
[da^a* 1 + C 2 al 1 - 1 a x A 2+2 for k = 2, 

for some constants 61,6*2 G C. This concludes the proposition. □ 



4.3.3 The embeddings into the space (III) 

We consider the embeddings of Xk,x (k = 1,2) into the spaces type (III^) (i = 
1, . . . ,8) which are classified in Proposition 14. 101 These spaces are isomorphic 

*° ^x((o £1 e 2 £3))(^ r \ < ^/^' ^ fe (-^))- We cons ider the image of this space by 
the map Smj) defined in Section l4~2l Here 



(ei,£2,£3) = 



'(1,1,1) 


if the 


case 


(nil), 


(1,1,0) 


if the 


case 


(ni 2 ), 


(1,0,1) 


if the 


case 


(nis), 


( (0,1,1) 


if the 


case 


(nil), 


(1,0,0) 


if the 


case 


(His), 


(0,1,0) 


if the 


case 


(III 6 ), 


(0,0,1) 


if the 


case 


(III7), 


,(0,0,0) 


if the 


case 


(III 8 ). 



We denote the image of 3(jtj) by 

^o.o^J^ 1 ^) = 3 (m) (C^ (0i0i0)ei)e2)e3)) (Ar\G/if; 4(A))). 

Proposition 4.30. The space C^ (0 e e A A; /fc(A)) is equal to the solution 
space of the following system of the differential equations on C°° (A) . 
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[0„i - (Ai + A 2 + k - 3)d ai + (2ttV^I— ) 2 £i 

«i (4.63) 

- (tfoa + #a 3 + t?«u) + Ai(A 2 + fc)]0 = 0, 

£l 2 7 rV Z T— + ^a 2 - (Ai + A 2 + fc - 3))0 = 0, (4.64) 
ai 

£i£ 2 </> = 0, (4.65) 

[i?a 2 - (Ax + A 2 + k - 2)tf a2 +(2W Z l-) 2 £i + (27^=1— ) 2 £ 2 

ai «2 (4.66) 

-(l?as+^ 4 ) + Al(A2 + fc)]^ = 0, 

e^W^I— (tf a2 + # 03 - (Ai + A 2 + fe - 2))0 = 0, (4.67) 

£ 2 £30 = 0, (4.68) 

(4.69) 



K 3 ~ (Ai + A 2 + k - l)tf a3 + (2^1 — ) 2 £ 2 

a 2 



+ (2^V^I— ) 2 £ 3 - tf„ 4 + Ai(A 2 + fe)]0 = 0, 
^ttV^I— (0«3 + ? ? a4 - (A! + A 2 + k - l))(f> = 0, (4.70) 
[< ~ (Ai + A 2 + k)i9 ai + (2TrV^l—) 2 e 3 + Ai(A 2 + k)]<j> = 0, (4.71) 

[0 O1 + <?a 2 + <?a 3 + ^a 4 - fcAl - (4 - fc)A 2 ]0 = 0. (4.72) 

Here <t> € C*°°(A). 

Proof. As well as Proposition 14.171 and Proposition 14.261 this system of differ- 
ential equations are obtained by the direct computation by using Lemma I4.12[ 
Proposition 14. 131 and Proposition 14. 161 □ 

Theorem 4.31. The space C'^ > (0 ei E2 B3 j (A; /^(A)) are written as follows. 

(i) When 

((1.1.1) 

(£i,£2,£3) = < (1, 1,0) , 

1(0,1,1) 

then we have 

(ii) When (£i,£ 2 ,£ 3 ) = (1,0,1), the space C~ (M t 1} (A; 4(A)) consists of 
followings. If k = 1, we ftave C^ (o o o x o (A; l|(A)) = {0}. If k = 2, the 
space C^ (0 0i0 _ 101) (A;/ 2 (A)) consists o/ 

l A!+A 2 -l 1 A t +A 2 + 1 

xj 2 ^ 2 x|a; 4 2 /(27ra;i)g(27ra;3) 

/or f(x),g(x) E aJl ( B( Al ~^ 2 ~ 2 ;a:). ffere we put 

a 2 a 4 
xi = — , xi = aia 2 , a; 3 = — , #4 = fl3 a 4- 
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Thus we have dimr C°° (A: /2(A)) = 4. T/iere is a 1- dimensional 

Xi(o, 0,0, 1,0,1) v ' v 

subs-pace of C^ (o o o 1 o (A; /2(A)) w/wc/i consists slowly increasing func- 
tions on { (x\ , . . . , x 4 ) I Xi G K>o , i = 1, . . . , 4} . 

(mj W-Tien (ei,£ 2 ,£3) = (1,0,0), t/ie space C^ (0>0 01 0>0) iA 4(A)) consists of 
1 A 1+ ^+ fc -3 Jx A2+1 x A2+1 «/fc=l, 

X>X 2 - /^jX j^ + l^l+^-1^+2 t/fc = 2; 

/or /(x) G £gt!B( Al ~ A | +fc ~ 4 ;a;) and Ci, C 2 G C. ffere we put 
02 

xi = — , x 2 = aia 2 , x 3 = a 3 , x 4 = 04. 
ai 

Thus if k = 1, we Ziaue dime C?° {A:I\(\)) = 2. There is a 1- 

J ' ^ X((o, 0,0, 1,0,0) v ' lv >' 

dimensional subspace of C^ (o o o i o o (A; 4(A)) which consists slowly in- 
creasing functions on {(xi, . . . , X4) \ Xi € K>o, i = 1, . . . ,4}. 
Also if k = 2. we ftaue dim c C°° (A:I 2 (X)) = 4. There is a 2- 

J 7 ^ Xi(o, 0,0, 1,0,0) v ' 

dimensional subspace of C^ i(o Q o 1 o o (A; /2(A)) which consists slowly in- 
creasing functions on {(x\, . . . ,x 4 ) I Xi € R>o,i = 1,. ..,4}. 

(iv) When {e 1} e 2 ,e 3 ) = (0,1,0), the space C^ [(o o o o i o) (A; 4(A)) consists of 
followings. We put 



Ifk = l, 



a-3 

x\ = ai,x 2 = — ,x 3 = a 2 a 3 , x 4 = a 4 . 
a 2 



1 A! + A 2 + fc^2 

x A2 x|x 3 2 x A2+1 /(27rx 2 ) 



/or /(x) e 0Jt03( A i^^; x). T/ms we Aave dim c C~ (0>0 1 0) (A; 4(A)) = 
2. There is a 1-dimensional subspace of 1 0) {A\ 4(A)) w/iic/i con- 
sists slowly increasing functions on {(x\, . . . , X4) | Xj e M>o, i = 1, • • • , 4}. 
Also if k = 2, 

1 A 1 + A 2 + fc-2 1 A t +A 2 + fc-2 

Cx Al x|x 3 2 x A2+ 2 ff i(27rx 2 ) + C"xi 2 x|x 3 2 x Al .g 2 (27rx 2 ) 

/or C,C" e C, gi G gn*B( A i-^- 3 ;x) and .a 2 € 07tO3( Aj ^ 2 -; x). 77ms 
we have dime C^ (0 1 o> ^ 2 ( A )) = ^- There is a 2-dimensional sub- 
space of C^ (o Q Q i o o (A; 4(A)) which consists slowly increasing functions 
on {(xi, . . . ,x 4 ) I Xi G R> ,« = 1, ... ,4}. 

^ ^hen (0i,0 2 ,0 3 ) = (0,0,1), tfre space C» wA01) (A;4(A)) consists of 

1 a 1+ a 2+ ^i fx A2 x A2 ifk = l, 

^ 4 2 3(2 ^ 3)X \c 1 x A2 .x A 1 - + C2X A ?x A2 ^ tf* = 2, 

forg(x) G 07103 ( A ^| 2 -^; x) and Ci,C 2 G C. Fere we put 

a 4 

xi = ai, x 2 = a 2 , x 3 = — , x 4 = a 3 a 4 . 

«3 
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Thus if k = 1, we have dinar C?° (j4;Ji(A)) = 2. There is a 1- 

1 ' ^ X!(0, 0,0, 0,0,1) v ' x \ // 

dimensional subspace of o o Q x (A; Ii(X)) which consists slowly in- 

creasing functions on {(xi, . . . , £4) \ Xi S R>o, i = 1, . . . ,4}. 
,4Zso if k = 2, we have dim c C°° M;/ 2 (A)) = 4. There is a 2- 

1 ' ^ X!(0, 0,0, 0,0,1) v ' iv 

dimensional subspace of C^ (o Q o o o (A; /2(A)) which consists slowly in- 
creasing functions on {{x\ 1 . . . , £4) | 2^ G R>o, t = 1, . . . , 4}. 

fwj Mien (£i,£ 2 ,£ 3 ) = (0,0,0), £/ie space C~ (o o o o o o) (A; 4(A)) consists of 
the followings. If k = 1, 

Oia^ a 2 a 3 a 4 ' ^2 a i a 2 a 3 a 4 

forQ eC,t=l,...,4, 
,4Zso i/ fc = 2, 

A 2 A 2 Ai Ai , f-i _Ai — 2 _Ai — 1 A 2 + l Ai 

Oia^ a 2 03 04 + ^2 a \ a 2 a 3 a 4 

i/o Ai-2 A 2 + l A 2 +l Aj , r „A 2 Ai-1 Ai-1 A 2 +2 
+ 030^ d 2 a 3 a 4 ~r 040-^ a 2 03 GS4 

, ^ Ai-2 A 2 + l Ai-1 A 2 +2 , r Ai-2 Ai-2 A 2 +2 A 2 +2 
-+- Osa^ a 2 03 a 4 + L'6 a i a 2 a 3 a 4 

/or Ci <G C, i = 1, . . . , 6. 

Proof, (i) It is immediately follows from equations (|4.65|) and (|4.68l) . 

(ii) If we put x\ = ^,X2 = aia 2 ,X3 = = a3»4, differential equations 
are written as 

[< - ^-^(Ai + A 2 + fc - 3)(Ai + A 2 + fc - 1) 

+ (2tt^1 Xi ) 2 + (Ax + fc - 3)(A 2 + l)]<f> = 0, 
[2^ 2 - (Ai + A 2 + k - 3)]0 = 0, 

[4l„ ~ tfx 3 - |((Ai + A 2 + fc) 2 + 1) + (2nV^lx 3 ) 2 + Ai(A 2 + fc)]</> = 0, 

[2^ 4 - (Ai + A 2 + k - l)}<f> = 0, 
[2d X2 + 2§ Xi - fcAi - (4 - k)\ 2 ]<f> = 0. 

Then we can solve these equations. 

(iii) If we put x\ — ^-,x% — aia 2 ,X3 = 03,2:4 = 04, differential equations 
are written as 

[< - ^i-J(Ai + A 2 + fc - 3)(Ai + A 2 + fc - 1) 

+ (2t:V^1x 1 ) 2 + (Ax + fc - 3)(A 2 + 1)]0 = 0, 
[20* a - (Ai + A 2 + k - 3)]<f> = Q, 
(i?x 3 - (Ai + fc - 3))(i? X3 - (A 2 + 1))0 = 0, 
(^ 4 -Ai)(^ 4 -(A 2 + fc))^ = 0, 
[0x, + ^ 4 + (1 - fc)Ai + (fc - 3)A 2 + (fc - 3)]0 = 0, 
[20^ + 0,3 + tf X4 - fcAi - (4 - fc)A 2 ]<£ = 0. 
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Then we can solve these equations. 

(iv) If we put x\ = a\,X2 = ^, x 3 = 0,20.3, %4 = 04, differential equations 
are written as 



-(Ai + (fc-4)))(i? Xl -A 2 )^) = 0, 
[< - <?* 2 - |(Ai + A 2 + fc - 2)(Ai + A 2 + fc) 

+ (27iV=Ia; 2 ) 2 - tf X4 + Ai(A 2 + h)]<j> = 0, 
(2i? X3 - (Ai + A 2 + k - 2))<j> = 0, 
(0* 4 -Ai)(0 X4 -(A 2 + fc))0 = O, 
[0 Xl + 2i? X3 + tf X4 - fcAi - (4 - k)\ 2 ]<t> = 0. 

Then we can solve these equations. 

(v) If we put x\ = ai, x 2 = a 2 , x 3 = ^,^4 = 03^4, differential equations arc 
written as 

(<?x 1 -(Ai-4 + A))(i? ai -A2)0 = O ) 
(tf X2 - (Ax - l))(0 Xa - (A 2 — 1 + = 0, 

- 0*s - J((Ai + A 2 + fc) 2 + 1) + (2^\/^T.x 3 ) 2 + Ai(A 2 + k)]<P = 0, 

[2^ 4 - (Ax + A 2 + k - 1)]0 - 0, 
[0 Xl + tf X2 + 2tf X4 - k\ 1 - (4 - fc)A 2 ]0 = 0. 

Then we can solve these equations. 

(vi) Differential equations are written as 

(^a 1 -(Ai-(fc-4)))(i? ai -A 2 )</» = 0, 
[< - (Ai + A 2 + k - 2)tf Q2 - (tf a3 + tf a J + Ai(A 2 + k)]<f> = 0, 
K - (Ai + A 2 + fc - l)i? a3 - d ai + Ai(A 2 + fc)]tf = 0, 
(tfa 4 -A!)(tf a4 -(A 2 + £))</> = 0, 

[0 Ol + #a 2 + 0„, + #a 4 - fcAl - (4 - fc)A 2 ]^ = 0. 

Then we can solve these equations. 

□ 



Appendix 

A The multiplicity one theorem for Horn's hy- 
pergeometric functions 

We will give a kind of the multiplicity one theorem for Horn's hypergeometric 
functions for the purpose of the application to the multiplicity theorem for the 
generalized Whittaker models. 

Let Pi(x) and Qi(x) are nonzero polynomials on x = (xi, . . . , x n ) for i = 
1, . . . , n. Then the Horn's hypergeometric functions are the solutions of the 
system of linear partial differential equations 

[x i P i (-d)-Q i (#)}f(x) = 0, i = l,...,n. (A.l) 
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Here — Xi-^- and d = (i9j, . . . , We assume that Pi and Q,; can be 
decomposed by products of linear factors, i.e., 

p <? 
Pi(s) = Y[((A k , s) - a), Qi(s) = l[((B h s) - dt) 

fc=i 1=1 

for s e R™, Ak,Bi S R™, Cfc,c?; G C and ( , ) denote the natural inner product 
in R™. We also assume Pi(s), Qi(s + e^) are relatively prime for i = 1, . . . ,n. 
Here = (0, . . . , 0, 1, 0, . . . , 0) (1 in the ith position). 

We consider the following system of difference equations associated with this 
system of differential equations (jA.ip , 

P l (~(s + e l ))<ft{s + e t ) = Qi{-s)(j){s) i = l,...,n. (A.2) 

We consider the general solutions for this system of difference equations (|A.2[) . 

Remark A.l. Let (ft be a solution of the system of difference equations iA.2fy . 
We consider an integral representation of Mellin-Barnes type, 

f{x) = [ cft(s)x- s ds. 



Then under the following assumptions, we can see that f{x) is a solution of 
the system of differential equations (A.l)) . 

A. For any i = 1, . . . ,n, the translation of the contour C with respect to 
the basis is homologically equivalent to C in the complement of the set of the 
singularities of the integrand <ft{s) in C n . 

B. The integral converges absolutely and it can be differentiated with respect 
to x sufficiently many times. 



If we put 



Pi{-{s + a)) 



Theorem A.2 (Orc-Sato-Sadykov [18] , [25] , [24] ) • The system of difference equa- 
tions (A.2))) is solvable if and only if 

Riis + e^Rjis) = Rj(s + ei)Ri(s), i,j = l,...,n. (A.3) 

If the system I A. 2(1 is solvable, then its solution is unique up to an arbitrary 
periodic function ip(s) with respect to ei, i.e., tft(s + ef) = ip(s), i — 1, . . . ,n. 
Furthermore, there exist p',q' G N, A' k , B[ e K" (1 < k < p' , 1 < I < q'), 
c' k ,d[ € C (1 < k < p , 1 < I < q ) and ti € R (i = l,...,n) such that the 
general solution of y£jQ is written as follows, 



n;.if(K,.)-o 1 



where t s = t 1 Sl • • ■t n Sn and tft{s) is an arbitrary periodic function satisfying 
ip(s + ei) = ip(s). 
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Suppose that the system of difference equations (|A.2p is solvable, i.e., the 
condition (|A.3|) is satisfied and we can choose a solution, 

\ I T-rp' 



which satisfies following conditions; 
C. we have the inequality, 



j2\{Bl,s)\-J2\{A' k , S )\>J2\si 



1=1 k=l i=l 

for seR" 

D. the function c/>(s) has no zero if each Re(si) are sufficiently large for 
i = 1, . . . , n. 

Remark A. 3. We consider the integral 

f(x)= / ... / ^{ S )x~ s d Sl 

J (J\ — v^— Too J (7 n — \/ — loo 

for appropriate o~i € K i = 1, . . . , n. Under the assumption C, it follows that the 
integral is absolutely convergent in the set {x £ R n | (tiXi, . . . , t n x n ) £ (R>o) n }. 

The following theorem is a generalization of the theorem of Diaconu and 
Goldfeld (Theorem 6.1.6 in 0) 

Theorem A. 4 (Multiplicity one). If f(x) is a solution of the system of Horn's 
hypergeometric differential equations (012]) which satisfies the growth condition 

sup \x a f(tx)\ < +oo 

ze(R>o)" 

for sufficiently large integers Ui £ N, i = l,...,n, then it is unique up to 
constant multiple. Here x a = x" 1 ■ ■ ■ x^ and tx = (iiii, . . . , t n x n ). 

Proof. We consider the Mellin transform of f(tx) as the function of x, 



poo 

M[f,s] = / 
Jo 



/(ta)^" 1 dx. 







This integral converges absolutely and M. [/, s] is analytic function of s if each 
Re(si) are sufficiently large by the assumption of f(x). Changing the variables 
x to tx = (tiXi, • • • , t n x n ), then we have 



oo ft oo 

s-l 



M[f,s]=t- S ■■• / f{x)x s - L dx. 

Jo Jo 



By the growth condition of f{x), we have 



pt 1 OO 
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by integration by parts for i = 1, . . . , n. Recall that f(x) satisfies the system of 
the partial differential equations (jA.f [) , then we have the system of the difference 
equations for M [/, s] , 

Pi(-(s + ei))M[f, a + <n] = Qi{-s)M[f, s] i = l,...,n. 

Hence by Theorem I A. 21 there is a periodic function ip( s ) an( i we have 

ULWA^V m _ !'"'°° }( x ) x '-^ dx. (A .4) 



IE,ir(Mt,»)-«l) 

By Stirling's formula and the assumption C, we obtain the estimate for Re(sj) > 

(i = l,...,n), 

Also by the Riemann-Lebesgue theorem, we have 

71 

M[f,s]^0 as 5^|Im(«i)|-f +oo. 

i=l 

Combining these estimates, we obtain the asymptotic behaviour of the periodic 
function 

V>(s)=0(exp(±7r|Im(*)l)), (A.5) 

as Im(s,) — > oo and the other Sj (i ^ j) are fixed. The right hand side of 
the equation (|A.4jl is the analytic function of s when Re(sj) (i — 1, . . . , n) are 
sufficiently large. Thus if we recall that the assumption D and the periodicy of 
i/j{s), we can see that f/K 5 ) is an entire function. We put zj = exp 2tt\/— 1 Si for 

1 = 1, . . . ,n. And we consider the Laurant expansion of <fr(s) with respect to z\, 

oo 



lK*) = ^ 4 1) ( S2 ,...,s„)z 1 fe . 



k=—oo 



Here cj^(s 2 , . . . , s n ) are periodic and entire functions for (s2, . . . , s n ) G C™ 1 . 
We write s$ = cr^ + \/— fr^ for e^, t; £l, i = f , . . . , n. We consider an integration 



„1 oo 

/ \ip(s)\ 2 dai = V |4 1) (s 2 ,--.,s„)| 2 exp(-47rfcr i 



fe= 

> |c| 1) (s 2 ,...,s„)| 2 exp(-47rfT i ) 

for every £ = 0, ±1, ±2, However the estimate (IA.5I) tells us that there exist 

constants M< € R>n and we have 



exp(7r|ri|) > Mi [ \tp{s)\ 2 dai 
Jo 



for sufficiently large 7"$. Thus we have (s 2 , . . . , s n ) — for t — ±1, ±2, . . .. 
The remaining coefficient (s 2 , . . . , s n ) is also the periodic and entire functions 
for (s 2 , . . . , s n ) € C I1_1 . Hence we apply the same argument for Cq 1 ' ) (s 2 , . . . , s n ) 
with respect to s 2 . And also we can succeed inductively for i = 3, . . . ,n. Thus 
we can conclude i/j(s) must be a constant. This completes the proof of the 
theorem. □ 
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B Horn's hyper geometric function H 



10 



We will give some facts about Horn's two variables hypergeometric function 
Hio. Horn's hypergeometric function Hio is the hypergeometric seires defined 
as follows, 



H 10 (a,d; x,y) = 



(a)2m-n x myU 



^— ' (d) m m\n\ 

m=0,n=0 v ' 

Here the symbol (a) m means the Pochhammer symbol, i.e., (a) m = 0(0 + 
1) • • • (a + (m — 1)) for a £ C and m £ N. It is not hard to see that this power 
series satisfies the system of hypergeometric partial differential equations, 

{x(2ti x - 0„ + o)(20 a - y + a + 1) - 0,(0, + d - l)}^(ar, y) = 0, 

{y-^(2^-^ + a)}0( a; ,y) =0. L> " ' 

It is known that the dimension of the solution space is 4 (cf. [2]). We define 
another convergent series 

00 / i\m+2n 

W 10 (a,d;x,y) = £ 1 j 

m— 0,n— 

Then the basis of the solution space are written by the power series below, 

Hi (a,d; x,y) 
y- d+1 E 10 (a-2d + 2,-d + 2;x,y), 
.T a Hi (a,d; x,x 2 y), 
x Q ^ d+1 Hio(a - 2d + 3, -d + 2 ; x, x 2 y). 



The system of hypergeometric differential equations (|B.1|I has the solution 
which has the Mellin-Barnes integral representation. This is written as follows, 

<t>( x >v) = 

f(?i-\-y/—l0Q /'(T2+V / — loO 

/ _ / _ r(s 1 )r(s 1 -2 S2 -a)r(s 2 )r(s 2 -d+l)(-x)- Sl y- S2 ds 1 ds 2 . 

J (Ji — \/ — 100 J (71 — \J — lOO 

Here o"i € K and 172 G K. satisfy the conditions, u\ > 0, 02 > max{0, Rc(o! — 1)} 
and (Ti — 2cr 2 > Re(a). This integral converges absolutely for x £ R<o and 

y e M> . 

Theorem B.l. If f(x,y) is a solution of the system W.l]) which satisfies that 

sup \x^y a2 f(-x,y)\ < +00 

x,2/GR> 

for sufficiently large a\,a 2 £ N, £/ien 

r<Ti + v — loo /*(T2 + \/ —loo 



/•CTli-V - -LOO ro2n~V — J-OO 

C / _ / _ r(si)r( Sl -2s2-a)r(s 2 )r(s2-rf+l)(-a;) _;!l ^ S2 dsids 

J (T\ — \f — lOO J (T2 — V — 1°° 

/or some constant C. 
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Proof. For 



C&1 1°° f^i-Vy/ —loo 



/ _ / _ r(s 1 )r(s 1 -2s 2 -a)r(s 2 )r(s 2 -d+i)(-x)- Si 2/ - S2 ds 1 d S2 , 

J ai — \/— lOO J (To — \/ — loo 



it is easy to see that satisfies the assumptions of Theorem IA.4I Hence we 
only need to check that <j> satisfies the growth condition. If we write a complex 
number s = a + y/—lr, we have 



Thus we have the inequality, 

\(j>{x,y)\ <M\x\~^\y\ 
for x G M<o and y € M>o- Here the constant 

p(Ti +\/ — lCJO /■<T2 + \/— lOO 
/ (Tl — \/— 1CX) J (72 — \/—l00 



M = 



f (Tl + V— TOO /• CT2 + \/— lOO 

/ _ / _ r( Sl )r( Sl -2s 2 -a)r( S2 )r(s 2 -rf + i)ds 1 ds 2 

^ (Tl — \/ — lOO ^ (7"9 — \/~ TOO 



We can choose <Ti and tr 2 as ci > 0, ct 2 > max{0,Re(d — 1)} and tri — 2ct 2 > 
Re(a). Thus 4>{x : y) satisfies the growth condition. □ 
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